{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "Congratulations for completing the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program!  In this notebook, you will learn how to control an agent in a more challenging environment, where the goal is to train a creature with four arms to walk forward.  **Note that this exercise is optional!**\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Crawler.app\"`\n",
    "- **Windows** (x86): `\"path/to/Crawler_Windows_x86/Crawler.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Crawler_Windows_x86_64/Crawler.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Crawler_Linux/Crawler.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Crawler_Linux/Crawler.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Crawler.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Crawler.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: CrawlerBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 129\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 20\n",
      "        Vector Action descriptions: , , , , , , , , , , , , , , , , , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='Crawler.exe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 12\n",
      "Size of each action: 20\n",
      "There are 12 agents. Each observes a state with length: 129\n",
      "The state for the first agent looks like: [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  2.25000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  1.78813934e-07  0.00000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093168e-01 -1.42857209e-01 -6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339906e+00 -1.42857209e-01\n",
      " -1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093347e-01 -1.42857209e-01 -6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339953e+00 -1.42857209e-01\n",
      " -1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093168e-01 -1.42857209e-01  6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339906e+00 -1.42857209e-01\n",
      "  1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093347e-01 -1.42857209e-01  6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339953e+00 -1.42857209e-01\n",
      "  1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "#states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "#scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "#while True:\n",
    "#    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "#    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "#    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "#    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "#    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "#    dones = env_info.local_done                        # see if episode finished\n",
    "#    scores += env_info.rewards                         # update the score (for each agent)\n",
    "#    states = next_states                               # roll over states to next time step\n",
    "#    if np.any(dones):                                  # exit loop if episode finished\n",
    "#        break\n",
    "#print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  collections  import deque\n",
    "from itertools import count\n",
    "import torch\n",
    "import time\n",
    "#from ddpg_agent import Agent\n",
    "from ppo_agent import Agent\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=8,\\\n",
    "              n_agent=num_agents, fc1_units=1024, fc2_units=1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1, Score: 48.98, Max: 48.98, Min: 48.98, T-max: 999   \n",
      "Episode: 2, Score: 47.49, Max: 48.98, Min: 47.49, T-max: 999   \n",
      "Episode: 3, Score: 48.12, Max: 48.98, Min: 47.49, T-max: 999   \n",
      "Episode: 4, Score: 47.23, Max: 48.98, Min: 47.23, T-max: 999   \n",
      "Episode: 5, Score: 47.23, Max: 48.98, Min: 47.23, T-max: 999   \n",
      "Episode: 6, Score: 47.15, Max: 48.98, Min: 47.15, T-max: 999   \n",
      "Episode: 7, Score: 55.17, Max: 55.17, Min: 47.15, T-max: 999   \n",
      "Episode: 8, Score: 54.20, Max: 55.17, Min: 47.15, T-max: 999   \n",
      "Episode: 9, Score: 52.36, Max: 55.17, Min: 47.15, T-max: 999   \n",
      "Episode: 10, Score: 55.42, Max: 55.42, Min: 47.15, T-max: 999   \n",
      "*** Episode 10 \t Average Score (over agents): 55.42 \t Average Score on 100 Episode: 50.34, Time: 00:02:44***\n",
      "Episode: 11, Score: 55.93, Max: 55.93, Min: 47.15, T-max: 999   \n",
      "Episode: 12, Score: 54.85, Max: 55.93, Min: 47.15, T-max: 999   \n",
      "Episode: 13, Score: 56.43, Max: 56.43, Min: 47.15, T-max: 999   \n",
      "Episode: 14, Score: 60.90, Max: 60.90, Min: 47.15, T-max: 999   \n",
      "Episode: 15, Score: 59.21, Max: 60.90, Min: 47.15, T-max: 999   \n",
      "Episode: 16, Score: 61.13, Max: 61.13, Min: 47.15, T-max: 999   \n",
      "Episode: 17, Score: 60.39, Max: 61.13, Min: 47.15, T-max: 999   \n",
      "Episode: 18, Score: 61.89, Max: 61.89, Min: 47.15, T-max: 999   \n",
      "Episode: 19, Score: 64.47, Max: 64.47, Min: 47.15, T-max: 999   \n",
      "Episode: 20, Score: 65.94, Max: 65.94, Min: 47.15, T-max: 999   \n",
      "*** Episode 20 \t Average Score (over agents): 65.94 \t Average Score on 100 Episode: 55.23, Time: 00:05:00***\n",
      "Episode: 21, Score: 63.50, Max: 65.94, Min: 47.15, T-max: 999   \n",
      "Episode: 22, Score: 62.90, Max: 65.94, Min: 47.15, T-max: 999   \n",
      "Episode: 23, Score: 66.58, Max: 66.58, Min: 47.15, T-max: 999   \n",
      "Episode: 24, Score: 65.11, Max: 66.58, Min: 47.15, T-max: 999   \n",
      "Episode: 25, Score: 66.09, Max: 66.58, Min: 47.15, T-max: 999   \n",
      "Episode: 26, Score: 67.56, Max: 67.56, Min: 47.15, T-max: 999   \n",
      "Episode: 27, Score: 65.01, Max: 67.56, Min: 47.15, T-max: 999   \n",
      "Episode: 28, Score: 70.30, Max: 70.30, Min: 47.15, T-max: 999   \n",
      "Episode: 29, Score: 69.55, Max: 70.30, Min: 47.15, T-max: 999   \n",
      "Episode: 30, Score: 71.41, Max: 71.41, Min: 47.15, T-max: 999   \n",
      "*** Episode 30 \t Average Score (over agents): 71.41 \t Average Score on 100 Episode: 59.08, Time: 00:07:18***\n",
      "Episode: 31, Score: 73.64, Max: 73.64, Min: 47.15, T-max: 999   \n",
      "Episode: 32, Score: 73.09, Max: 73.64, Min: 47.15, T-max: 999   \n",
      "Episode: 33, Score: 72.32, Max: 73.64, Min: 47.15, T-max: 999   \n",
      "Episode: 34, Score: 75.52, Max: 75.52, Min: 47.15, T-max: 999   \n",
      "Episode: 35, Score: 79.04, Max: 79.04, Min: 47.15, T-max: 999   \n",
      "Episode: 36, Score: 78.72, Max: 79.04, Min: 47.15, T-max: 999   \n",
      "Episode: 37, Score: 69.82, Max: 79.04, Min: 47.15, T-max: 999   \n",
      "Episode: 38, Score: 65.80, Max: 79.04, Min: 47.15, T-max: 999   \n",
      "Episode: 39, Score: 80.58, Max: 80.58, Min: 47.15, T-max: 999   \n",
      "Episode: 40, Score: 79.58, Max: 80.58, Min: 47.15, T-max: 999   \n",
      "*** Episode 40 \t Average Score (over agents): 79.58 \t Average Score on 100 Episode: 63.02, Time: 00:09:32***\n",
      "Episode: 41, Score: 77.20, Max: 80.58, Min: 47.15, T-max: 999   \n",
      "Episode: 42, Score: 78.64, Max: 80.58, Min: 47.15, T-max: 999   \n",
      "Episode: 43, Score: 77.93, Max: 80.58, Min: 47.15, T-max: 999   \n",
      "Episode: 44, Score: 82.42, Max: 82.42, Min: 47.15, T-max: 999   \n",
      "Episode: 45, Score: 83.63, Max: 83.63, Min: 47.15, T-max: 999   \n",
      "Episode: 46, Score: 84.08, Max: 84.08, Min: 47.15, T-max: 999   \n",
      "Episode: 47, Score: 83.66, Max: 84.08, Min: 47.15, T-max: 999   \n",
      "Episode: 48, Score: 89.49, Max: 89.49, Min: 47.15, T-max: 999   \n",
      "Episode: 49, Score: 90.72, Max: 90.72, Min: 47.15, T-max: 999   \n",
      "Episode: 50, Score: 93.84, Max: 93.84, Min: 47.15, T-max: 999   \n",
      "*** Episode 50 \t Average Score (over agents): 93.84 \t Average Score on 100 Episode: 67.24, Time: 00:11:45***\n",
      "Episode: 51, Score: 94.80, Max: 94.80, Min: 47.15, T-max: 999   \n",
      "Episode: 52, Score: 97.87, Max: 97.87, Min: 47.15, T-max: 999   \n",
      "Episode: 53, Score: 101.54, Max: 101.54, Min: 47.15, T-max: 999   \n",
      "Episode: 54, Score: 97.33, Max: 101.54, Min: 47.15, T-max: 999   \n",
      "Episode: 55, Score: 104.83, Max: 104.83, Min: 47.15, T-max: 999   \n",
      "Episode: 56, Score: 99.84, Max: 104.83, Min: 47.15, T-max: 999   \n",
      "Episode: 57, Score: 104.88, Max: 104.88, Min: 47.15, T-max: 999   \n",
      "Episode: 58, Score: 103.56, Max: 104.88, Min: 47.15, T-max: 999   \n",
      "Episode: 59, Score: 103.47, Max: 104.88, Min: 47.15, T-max: 999   \n",
      "Episode: 60, Score: 97.05, Max: 104.88, Min: 47.15, T-max: 999   \n",
      "*** Episode 60 \t Average Score (over agents): 97.05 \t Average Score on 100 Episode: 72.79, Time: 00:14:03***\n",
      "Episode: 61, Score: 115.15, Max: 115.15, Min: 47.15, T-max: 999   \n",
      "Episode: 62, Score: 113.26, Max: 115.15, Min: 47.15, T-max: 999   \n",
      "Episode: 63, Score: 110.56, Max: 115.15, Min: 47.15, T-max: 999   \n",
      "Episode: 64, Score: 124.10, Max: 124.10, Min: 47.15, T-max: 999   \n",
      "Episode: 65, Score: 115.44, Max: 124.10, Min: 47.15, T-max: 999   \n",
      "Episode: 66, Score: 120.75, Max: 124.10, Min: 47.15, T-max: 999   \n",
      "Episode: 67, Score: 130.18, Max: 130.18, Min: 47.15, T-max: 999   \n",
      "Episode: 68, Score: 128.74, Max: 130.18, Min: 47.15, T-max: 999   \n",
      "Episode: 69, Score: 129.37, Max: 130.18, Min: 47.15, T-max: 999   \n",
      "Episode: 70, Score: 140.48, Max: 140.48, Min: 47.15, T-max: 999   \n",
      "*** Episode 70 \t Average Score (over agents): 140.48 \t Average Score on 100 Episode: 79.93, Time: 00:16:16***\n",
      "Episode: 71, Score: 136.37, Max: 140.48, Min: 47.15, T-max: 999   \n",
      "Episode: 72, Score: 139.86, Max: 140.48, Min: 47.15, T-max: 999   \n",
      "Episode: 73, Score: 142.62, Max: 142.62, Min: 47.15, T-max: 999   \n",
      "Episode: 74, Score: 144.65, Max: 144.65, Min: 47.15, T-max: 999   \n",
      "Episode: 75, Score: 131.61, Max: 144.65, Min: 47.15, T-max: 999   \n",
      "Episode: 76, Score: 162.31, Max: 162.31, Min: 47.15, T-max: 999   \n",
      "Episode: 77, Score: 143.40, Max: 162.31, Min: 47.15, T-max: 999   \n",
      "Episode: 78, Score: 165.83, Max: 165.83, Min: 47.15, T-max: 999   \n",
      "Episode: 79, Score: 174.25, Max: 174.25, Min: 47.15, T-max: 999   \n",
      "Episode: 80, Score: 182.02, Max: 182.02, Min: 47.15, T-max: 999   \n",
      "*** Episode 80 \t Average Score (over agents): 182.02 \t Average Score on 100 Episode: 88.98, Time: 00:18:30***\n",
      "Episode: 81, Score: 178.30, Max: 182.02, Min: 47.15, T-max: 999   \n",
      "Episode: 82, Score: 185.10, Max: 185.10, Min: 47.15, T-max: 999   \n",
      "Episode: 83, Score: 214.48, Max: 214.48, Min: 47.15, T-max: 999   \n",
      "Episode: 84, Score: 189.03, Max: 214.48, Min: 47.15, T-max: 999   \n",
      "Episode: 85, Score: 207.73, Max: 214.48, Min: 47.15, T-max: 999   \n",
      "Episode: 86, Score: 207.93, Max: 214.48, Min: 47.15, T-max: 999   \n",
      "Episode: 87, Score: 209.89, Max: 214.48, Min: 47.15, T-max: 999   \n",
      "Episode: 88, Score: 202.24, Max: 214.48, Min: 47.15, T-max: 999   \n",
      "Episode: 89, Score: 212.43, Max: 214.48, Min: 47.15, T-max: 999   \n",
      "Episode: 90, Score: 219.10, Max: 219.10, Min: 47.15, T-max: 999   \n",
      "*** Episode 90 \t Average Score (over agents): 219.10 \t Average Score on 100 Episode: 101.61, Time: 00:20:46***\n",
      "Episode: 91, Score: 232.42, Max: 232.42, Min: 47.15, T-max: 999   \n",
      "Episode: 92, Score: 236.65, Max: 236.65, Min: 47.15, T-max: 999   \n",
      "Episode: 93, Score: 242.55, Max: 242.55, Min: 47.15, T-max: 999   \n",
      "Episode: 94, Score: 241.36, Max: 242.55, Min: 47.15, T-max: 999   \n",
      "Episode: 95, Score: 260.73, Max: 260.73, Min: 47.15, T-max: 999   \n",
      "Episode: 96, Score: 261.71, Max: 261.71, Min: 47.15, T-max: 999   \n",
      "Episode: 97, Score: 279.44, Max: 279.44, Min: 47.15, T-max: 999   \n",
      "Episode: 98, Score: 252.27, Max: 279.44, Min: 47.15, T-max: 999   \n",
      "Episode: 99, Score: 274.17, Max: 279.44, Min: 47.15, T-max: 999   \n",
      "Episode: 100, Score: 290.34, Max: 290.34, Min: 47.15, T-max: 999   \n",
      "*** Episode 100 \t Average Score (over agents): 290.34 \t Average Score on 100 Episode: 117.16, Time: 00:23:00***\n",
      "Episode: 101, Score: 277.11, Max: 290.34, Min: 47.15, T-max: 999   \n",
      "Episode: 102, Score: 281.88, Max: 290.34, Min: 47.15, T-max: 999   \n",
      "Episode: 103, Score: 340.83, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "Episode: 104, Score: 326.27, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "Episode: 105, Score: 312.54, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "Episode: 106, Score: 316.80, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "Episode: 107, Score: 327.66, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "Episode: 108, Score: 315.88, Max: 340.83, Min: 47.15, T-max: 999   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 109, Score: 333.93, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "Episode: 110, Score: 314.47, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "*** Episode 110 \t Average Score (over agents): 314.47 \t Average Score on 100 Episode: 143.60, Time: 00:25:13***\n",
      "Episode: 111, Score: 330.29, Max: 340.83, Min: 47.15, T-max: 999   \n",
      "Episode: 112, Score: 341.92, Max: 341.92, Min: 47.15, T-max: 999   \n",
      "Episode: 113, Score: 336.35, Max: 341.92, Min: 47.15, T-max: 999   \n",
      "Episode: 114, Score: 318.02, Max: 341.92, Min: 47.15, T-max: 999   \n",
      "Episode: 115, Score: 352.38, Max: 352.38, Min: 47.15, T-max: 999   \n",
      "Episode: 116, Score: 333.86, Max: 352.38, Min: 47.15, T-max: 999   \n",
      "Episode: 117, Score: 357.39, Max: 357.39, Min: 47.15, T-max: 999   \n",
      "Episode: 118, Score: 339.12, Max: 357.39, Min: 47.15, T-max: 999   \n",
      "Episode: 119, Score: 340.77, Max: 357.39, Min: 47.15, T-max: 999   \n",
      "Episode: 120, Score: 337.00, Max: 357.39, Min: 47.15, T-max: 999   \n",
      "*** Episode 120 \t Average Score (over agents): 337.00 \t Average Score on 100 Episode: 171.46, Time: 00:27:29***\n",
      "Episode: 121, Score: 356.57, Max: 357.39, Min: 47.15, T-max: 999   \n",
      "Episode: 122, Score: 363.09, Max: 363.09, Min: 47.15, T-max: 999   \n",
      "Episode: 123, Score: 370.39, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 124, Score: 353.76, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 125, Score: 358.19, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 126, Score: 356.12, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 127, Score: 324.75, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 128, Score: 337.11, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 129, Score: 344.38, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 130, Score: 337.99, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "*** Episode 130 \t Average Score (over agents): 337.99 \t Average Score on 100 Episode: 199.81, Time: 00:29:43***\n",
      "Episode: 131, Score: 343.86, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 132, Score: 341.06, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 133, Score: 368.24, Max: 370.39, Min: 47.15, T-max: 999   \n",
      "Episode: 134, Score: 383.04, Max: 383.04, Min: 47.15, T-max: 999   \n",
      "Episode: 135, Score: 372.58, Max: 383.04, Min: 47.15, T-max: 999   \n",
      "Episode: 136, Score: 363.92, Max: 383.04, Min: 47.15, T-max: 999   \n",
      "Episode: 137, Score: 334.60, Max: 383.04, Min: 47.15, T-max: 999   \n",
      "Episode: 138, Score: 354.48, Max: 383.04, Min: 47.15, T-max: 999   \n",
      "Episode: 139, Score: 384.95, Max: 384.95, Min: 47.15, T-max: 999   \n",
      "Episode: 140, Score: 370.04, Max: 384.95, Min: 47.15, T-max: 999   \n",
      "*** Episode 140 \t Average Score (over agents): 370.04 \t Average Score on 100 Episode: 228.49, Time: 00:31:56***\n",
      "Episode: 141, Score: 370.16, Max: 384.95, Min: 47.15, T-max: 999   \n",
      "Episode: 142, Score: 376.75, Max: 384.95, Min: 47.15, T-max: 999   \n",
      "Episode: 143, Score: 358.29, Max: 384.95, Min: 47.15, T-max: 999   \n",
      "Episode: 144, Score: 386.42, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 145, Score: 357.66, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 146, Score: 353.54, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 147, Score: 363.76, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 148, Score: 280.31, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 149, Score: 326.57, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 150, Score: 296.12, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "*** Episode 150 \t Average Score (over agents): 296.12 \t Average Score on 100 Episode: 254.77, Time: 00:34:12***\n",
      "Episode: 151, Score: 371.76, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 152, Score: 340.46, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 153, Score: 347.34, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 154, Score: 354.06, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 155, Score: 335.20, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 156, Score: 317.14, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 157, Score: 313.40, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 158, Score: 354.51, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 159, Score: 298.18, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 160, Score: 355.56, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "*** Episode 160 \t Average Score (over agents): 355.56 \t Average Score on 100 Episode: 278.60, Time: 00:36:25***\n",
      "Episode: 161, Score: 334.32, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 162, Score: 347.36, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 163, Score: 375.30, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 164, Score: 375.98, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 165, Score: 359.19, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 166, Score: 381.46, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 167, Score: 374.47, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 168, Score: 359.77, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 169, Score: 370.06, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 170, Score: 342.91, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "*** Episode 170 \t Average Score (over agents): 342.91 \t Average Score on 100 Episode: 302.52, Time: 00:38:39***\n",
      "Episode: 171, Score: 360.42, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 172, Score: 358.90, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 173, Score: 373.99, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 174, Score: 378.21, Max: 386.42, Min: 47.15, T-max: 999   \n",
      "Episode: 175, Score: 387.04, Max: 387.04, Min: 47.15, T-max: 999   \n",
      "Episode: 176, Score: 380.83, Max: 387.04, Min: 47.15, T-max: 999   \n",
      "Episode: 177, Score: 400.99, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 178, Score: 338.69, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 179, Score: 318.97, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 180, Score: 313.99, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 180 \t Average Score (over agents): 313.99 \t Average Score on 100 Episode: 323.41, Time: 00:40:55***\n",
      "Episode: 181, Score: 395.08, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 182, Score: 389.90, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 183, Score: 380.09, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 184, Score: 135.88, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 185, Score: 164.10, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 186, Score: 115.07, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 187, Score: 352.86, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 188, Score: 362.75, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 189, Score: 382.61, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 190, Score: 332.26, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 190 \t Average Score (over agents): 332.26 \t Average Score on 100 Episode: 333.26, Time: 00:43:09***\n",
      "Episode: 191, Score: 320.99, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 192, Score: 331.75, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 193, Score: 346.09, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 194, Score: 337.60, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 195, Score: 355.72, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 196, Score: 368.37, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 197, Score: 353.37, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 198, Score: 372.97, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 199, Score: 377.57, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 200, Score: 362.88, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 200 \t Average Score (over agents): 362.88 \t Average Score on 100 Episode: 342.82, Time: 00:45:22***\n",
      "Episode: 201, Score: 374.15, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 202, Score: 366.38, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 203, Score: 382.78, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 204, Score: 370.81, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 205, Score: 321.65, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 206, Score: 350.50, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 207, Score: 360.93, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 208, Score: 379.91, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 209, Score: 367.84, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 210, Score: 353.85, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 210 \t Average Score (over agents): 353.85 \t Average Score on 100 Episode: 347.63, Time: 00:47:39***\n",
      "Episode: 211, Score: 378.89, Max: 400.99, Min: 47.15, T-max: 999   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 212, Score: 385.54, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 213, Score: 376.65, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 214, Score: 363.47, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 215, Score: 391.35, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 216, Score: 375.04, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 217, Score: 362.66, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 218, Score: 375.63, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 219, Score: 364.19, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 220, Score: 361.11, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 220 \t Average Score (over agents): 361.11 \t Average Score on 100 Episode: 351.10, Time: 00:49:52***\n",
      "Episode: 221, Score: 361.24, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 222, Score: 373.98, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 223, Score: 376.55, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 224, Score: 388.57, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 225, Score: 387.85, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 226, Score: 383.75, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 227, Score: 378.83, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 228, Score: 375.05, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 229, Score: 372.69, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 230, Score: 366.08, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 230 \t Average Score (over agents): 366.08 \t Average Score on 100 Episode: 353.73, Time: 00:52:07***\n",
      "Episode: 231, Score: 365.94, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 232, Score: 367.67, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 233, Score: 350.18, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 234, Score: 360.64, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 235, Score: 381.28, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 236, Score: 366.58, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 237, Score: 357.57, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 238, Score: 376.89, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 239, Score: 359.00, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 240, Score: 356.15, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 240 \t Average Score (over agents): 356.15 \t Average Score on 100 Episode: 353.98, Time: 00:54:23***\n",
      "Episode: 241, Score: 354.35, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 242, Score: 380.78, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 243, Score: 374.36, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 244, Score: 366.37, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 245, Score: 380.33, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 246, Score: 374.18, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 247, Score: 378.39, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 248, Score: 348.56, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 249, Score: 362.93, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 250, Score: 326.79, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 250 \t Average Score (over agents): 326.79 \t Average Score on 100 Episode: 355.75, Time: 00:56:36***\n",
      "Episode: 251, Score: 359.13, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 252, Score: 344.85, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 253, Score: 367.65, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 254, Score: 357.36, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 255, Score: 350.02, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 256, Score: 368.75, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 257, Score: 392.83, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 258, Score: 379.45, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 259, Score: 368.39, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 260, Score: 365.40, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 260 \t Average Score (over agents): 365.40 \t Average Score on 100 Episode: 358.41, Time: 00:58:50***\n",
      "Episode: 261, Score: 386.26, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 262, Score: 374.93, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 263, Score: 396.80, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 264, Score: 382.22, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 265, Score: 369.13, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 266, Score: 371.32, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 267, Score: 354.84, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 268, Score: 395.91, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 269, Score: 397.45, Max: 400.99, Min: 47.15, T-max: 999   \n",
      "Episode: 270, Score: 412.08, Max: 412.08, Min: 47.15, T-max: 999   \n",
      "*** Episode 270 \t Average Score (over agents): 412.08 \t Average Score on 100 Episode: 360.62, Time: 01:01:12***\n",
      "Episode: 271, Score: 405.19, Max: 412.08, Min: 47.15, T-max: 999   \n",
      "Episode: 272, Score: 415.23, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 273, Score: 371.75, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 274, Score: 381.72, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 275, Score: 392.61, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 276, Score: 386.56, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 277, Score: 408.68, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 278, Score: 361.64, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 279, Score: 353.35, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 280, Score: 396.20, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "*** Episode 280 \t Average Score (over agents): 396.20 \t Average Score on 100 Episode: 363.22, Time: 01:03:27***\n",
      "Episode: 281, Score: 385.37, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 282, Score: 409.75, Max: 415.23, Min: 47.15, T-max: 999   \n",
      "Episode: 283, Score: 425.55, Max: 425.55, Min: 47.15, T-max: 999   \n",
      "Episode: 284, Score: 430.71, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 285, Score: 414.24, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 286, Score: 323.28, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 287, Score: 294.68, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 288, Score: 298.32, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 289, Score: 418.40, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 290, Score: 405.77, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "*** Episode 290 \t Average Score (over agents): 405.77 \t Average Score on 100 Episode: 371.18, Time: 01:05:42***\n",
      "Episode: 291, Score: 401.63, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 292, Score: 394.25, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 293, Score: 387.72, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 294, Score: 416.00, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 295, Score: 413.95, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 296, Score: 387.82, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 297, Score: 398.76, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 298, Score: 418.04, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 299, Score: 409.58, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 300, Score: 408.76, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "*** Episode 300 \t Average Score (over agents): 408.76 \t Average Score on 100 Episode: 376.27, Time: 01:07:59***\n",
      "Episode: 301, Score: 415.73, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 302, Score: 419.96, Max: 430.71, Min: 47.15, T-max: 999   \n",
      "Episode: 303, Score: 440.14, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 304, Score: 409.93, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 305, Score: 421.53, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 306, Score: 413.08, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 307, Score: 415.94, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 308, Score: 427.51, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 309, Score: 397.28, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 310, Score: 393.90, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "*** Episode 310 \t Average Score (over agents): 393.90 \t Average Score on 100 Episode: 381.53, Time: 01:10:19***\n",
      "Episode: 311, Score: 434.93, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 312, Score: 394.51, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 313, Score: 390.63, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 314, Score: 408.12, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 315, Score: 418.15, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 316, Score: 375.60, Max: 440.14, Min: 47.15, T-max: 999   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 317, Score: 399.81, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 318, Score: 390.62, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 319, Score: 433.96, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 320, Score: 437.85, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "*** Episode 320 \t Average Score (over agents): 437.85 \t Average Score on 100 Episode: 385.03, Time: 01:12:43***\n",
      "Episode: 321, Score: 436.05, Max: 440.14, Min: 47.15, T-max: 999   \n",
      "Episode: 322, Score: 450.24, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 323, Score: 433.97, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 324, Score: 413.95, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 325, Score: 443.08, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 326, Score: 424.44, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 327, Score: 391.77, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 328, Score: 403.66, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 329, Score: 377.42, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 330, Score: 439.43, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 330 \t Average Score (over agents): 439.43 \t Average Score on 100 Episode: 389.52, Time: 01:15:07***\n",
      "Episode: 331, Score: 439.87, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 332, Score: 427.66, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 333, Score: 420.32, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 334, Score: 421.84, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 335, Score: 433.28, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 336, Score: 445.31, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 337, Score: 438.72, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 338, Score: 426.57, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 339, Score: 438.34, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 340, Score: 420.91, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 340 \t Average Score (over agents): 420.91 \t Average Score on 100 Episode: 396.23, Time: 01:17:33***\n",
      "Episode: 341, Score: 400.68, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 342, Score: 434.83, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 343, Score: 439.75, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 344, Score: 444.87, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 345, Score: 420.08, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 346, Score: 406.71, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 347, Score: 378.21, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 348, Score: 405.29, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 349, Score: 429.22, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 350, Score: 399.48, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 350 \t Average Score (over agents): 399.48 \t Average Score on 100 Episode: 401.35, Time: 01:19:50***\n",
      "Episode: 351, Score: 392.11, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 352, Score: 413.23, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 353, Score: 433.15, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 354, Score: 396.85, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 355, Score: 416.56, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 356, Score: 423.66, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 357, Score: 381.91, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 358, Score: 403.66, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 359, Score: 419.93, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 360, Score: 392.26, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 360 \t Average Score (over agents): 392.26 \t Average Score on 100 Episode: 405.55, Time: 01:22:13***\n",
      "Episode: 361, Score: 405.79, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 362, Score: 438.38, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 363, Score: 421.10, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 364, Score: 391.95, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 365, Score: 394.31, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 366, Score: 418.04, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 367, Score: 413.02, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 368, Score: 416.96, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 369, Score: 406.17, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 370, Score: 383.93, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 370 \t Average Score (over agents): 383.93 \t Average Score on 100 Episode: 408.04, Time: 01:24:36***\n",
      "Episode: 371, Score: 379.16, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 372, Score: 388.63, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 373, Score: 407.98, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 374, Score: 402.11, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 375, Score: 394.49, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 376, Score: 411.88, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 377, Score: 409.63, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 378, Score: 407.33, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 379, Score: 377.57, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 380, Score: 398.40, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 380 \t Average Score (over agents): 398.40 \t Average Score on 100 Episode: 409.08, Time: 01:26:58***\n",
      "Episode: 381, Score: 370.65, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 382, Score: 407.64, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 383, Score: 412.80, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 384, Score: 405.16, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 385, Score: 384.90, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 386, Score: 402.68, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 387, Score: 397.31, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 388, Score: 409.18, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 389, Score: 400.72, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 390, Score: 425.88, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 390 \t Average Score (over agents): 425.88 \t Average Score on 100 Episode: 411.19, Time: 01:29:24***\n",
      "Episode: 391, Score: 405.59, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 392, Score: 416.07, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 393, Score: 420.67, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 394, Score: 414.53, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 395, Score: 409.03, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 396, Score: 392.27, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 397, Score: 434.97, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 398, Score: 450.05, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 399, Score: 417.89, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 400, Score: 369.32, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "*** Episode 400 \t Average Score (over agents): 369.32 \t Average Score on 100 Episode: 412.13, Time: 01:31:46***\n",
      "Episode: 401, Score: 336.46, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 402, Score: 348.16, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 403, Score: 426.38, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 404, Score: 426.87, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 405, Score: 414.30, Max: 450.24, Min: 47.15, T-max: 999   \n",
      "Episode: 406, Score: 452.86, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 407, Score: 436.94, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 408, Score: 448.85, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 409, Score: 436.91, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 410, Score: 439.51, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "*** Episode 410 \t Average Score (over agents): 439.51 \t Average Score on 100 Episode: 412.25, Time: 01:34:06***\n",
      "Episode: 411, Score: 438.90, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 412, Score: 421.97, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 413, Score: 430.74, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 414, Score: 414.52, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 415, Score: 452.51, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 416, Score: 443.79, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 417, Score: 439.48, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 418, Score: 414.67, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 419, Score: 415.68, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 420, Score: 432.47, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "*** Episode 420 \t Average Score (over agents): 432.47 \t Average Score on 100 Episode: 414.45, Time: 01:36:25***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 421, Score: 444.18, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 422, Score: 435.00, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 423, Score: 425.30, Max: 452.86, Min: 47.15, T-max: 999   \n",
      "Episode: 424, Score: 456.88, Max: 456.88, Min: 47.15, T-max: 999   \n",
      "Episode: 425, Score: 446.84, Max: 456.88, Min: 47.15, T-max: 999   \n",
      "Episode: 426, Score: 464.36, Max: 464.36, Min: 47.15, T-max: 999   \n",
      "Episode: 427, Score: 450.36, Max: 464.36, Min: 47.15, T-max: 999   \n",
      "Episode: 428, Score: 451.00, Max: 464.36, Min: 47.15, T-max: 999   \n",
      "Episode: 429, Score: 429.37, Max: 464.36, Min: 47.15, T-max: 999   \n",
      "Episode: 430, Score: 467.46, Max: 467.46, Min: 47.15, T-max: 999   \n",
      "*** Episode 430 \t Average Score (over agents): 467.46 \t Average Score on 100 Episode: 417.02, Time: 01:38:38***\n",
      "Episode: 431, Score: 459.15, Max: 467.46, Min: 47.15, T-max: 999   \n",
      "Episode: 432, Score: 469.07, Max: 469.07, Min: 47.15, T-max: 999   \n",
      "Episode: 433, Score: 448.91, Max: 469.07, Min: 47.15, T-max: 999   \n",
      "Episode: 434, Score: 451.97, Max: 469.07, Min: 47.15, T-max: 999   \n",
      "Episode: 435, Score: 457.03, Max: 469.07, Min: 47.15, T-max: 999   \n",
      "Episode: 436, Score: 463.45, Max: 469.07, Min: 47.15, T-max: 999   \n",
      "Episode: 437, Score: 478.71, Max: 478.71, Min: 47.15, T-max: 999   \n",
      "Episode: 438, Score: 454.82, Max: 478.71, Min: 47.15, T-max: 999   \n",
      "Episode: 439, Score: 478.47, Max: 478.71, Min: 47.15, T-max: 999   \n",
      "Episode: 440, Score: 455.39, Max: 478.71, Min: 47.15, T-max: 999   \n",
      "*** Episode 440 \t Average Score (over agents): 455.39 \t Average Score on 100 Episode: 420.06, Time: 01:40:51***\n",
      "Episode: 441, Score: 446.62, Max: 478.71, Min: 47.15, T-max: 999   \n",
      "Episode: 442, Score: 477.83, Max: 478.71, Min: 47.15, T-max: 999   \n",
      "Episode: 443, Score: 480.99, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "Episode: 444, Score: 462.52, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "Episode: 445, Score: 476.28, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "Episode: 446, Score: 454.80, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "Episode: 447, Score: 466.58, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "Episode: 448, Score: 473.45, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "Episode: 449, Score: 479.07, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "Episode: 450, Score: 474.72, Max: 480.99, Min: 47.15, T-max: 999   \n",
      "*** Episode 450 \t Average Score (over agents): 474.72 \t Average Score on 100 Episode: 425.40, Time: 01:43:07***\n",
      "Episode: 451, Score: 496.79, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 452, Score: 493.63, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 453, Score: 475.81, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 454, Score: 475.35, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 455, Score: 489.37, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 456, Score: 478.11, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 457, Score: 486.53, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 458, Score: 477.41, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 459, Score: 485.87, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 460, Score: 487.81, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "*** Episode 460 \t Average Score (over agents): 487.81 \t Average Score on 100 Episode: 433.13, Time: 01:45:21***\n",
      "Episode: 461, Score: 479.35, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 462, Score: 468.13, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 463, Score: 456.25, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 464, Score: 469.15, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 465, Score: 471.39, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 466, Score: 460.76, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 467, Score: 453.60, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 468, Score: 472.98, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 469, Score: 459.32, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 470, Score: 461.92, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "*** Episode 470 \t Average Score (over agents): 461.92 \t Average Score on 100 Episode: 438.77, Time: 01:47:34***\n",
      "Episode: 471, Score: 483.21, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 472, Score: 437.82, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 473, Score: 459.24, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 474, Score: 469.49, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 475, Score: 435.44, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 476, Score: 443.43, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 477, Score: 445.15, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 478, Score: 411.72, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 479, Score: 458.39, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 480, Score: 439.02, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "*** Episode 480 \t Average Score (over agents): 439.02 \t Average Score on 100 Episode: 443.82, Time: 01:49:50***\n",
      "Episode: 481, Score: 436.32, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 482, Score: 431.72, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 483, Score: 399.89, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 484, Score: 431.60, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 485, Score: 433.78, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 486, Score: 419.99, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 487, Score: 433.76, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 488, Score: 450.42, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 489, Score: 432.44, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 490, Score: 408.00, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "*** Episode 490 \t Average Score (over agents): 408.00 \t Average Score on 100 Episode: 446.43, Time: 01:52:03***\n",
      "Episode: 491, Score: 429.15, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 492, Score: 431.55, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 493, Score: 453.66, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 494, Score: 440.97, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 495, Score: 458.42, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 496, Score: 465.63, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 497, Score: 410.24, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 498, Score: 448.01, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 499, Score: 442.91, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 500, Score: 436.42, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "*** Episode 500 \t Average Score (over agents): 436.42 \t Average Score on 100 Episode: 449.30, Time: 01:54:17***\n",
      "Episode: 501, Score: 459.30, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 502, Score: 456.99, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 503, Score: 457.76, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 504, Score: 429.17, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 505, Score: 482.92, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 506, Score: 454.37, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 507, Score: 474.85, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 508, Score: 461.49, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 509, Score: 470.79, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 510, Score: 492.79, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "*** Episode 510 \t Average Score (over agents): 492.79 \t Average Score on 100 Episode: 454.03, Time: 01:56:32***\n",
      "Episode: 511, Score: 478.89, Max: 496.79, Min: 47.15, T-max: 999   \n",
      "Episode: 512, Score: 505.87, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 513, Score: 491.87, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 514, Score: 415.79, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 515, Score: 410.14, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 516, Score: 409.97, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 517, Score: 393.68, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 518, Score: 420.17, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 519, Score: 412.25, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 520, Score: 471.68, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "*** Episode 520 \t Average Score (over agents): 471.68 \t Average Score on 100 Episode: 455.09, Time: 01:58:46***\n",
      "Episode: 521, Score: 451.74, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 522, Score: 456.07, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 523, Score: 481.45, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 524, Score: 476.46, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 525, Score: 488.10, Max: 505.87, Min: 47.15, T-max: 999   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 526, Score: 464.34, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 527, Score: 458.67, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 528, Score: 468.51, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 529, Score: 495.23, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "Episode: 530, Score: 496.18, Max: 505.87, Min: 47.15, T-max: 999   \n",
      "*** Episode 530 \t Average Score (over agents): 496.18 \t Average Score on 100 Episode: 457.75, Time: 02:00:59***\n",
      "Episode: 531, Score: 508.95, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 532, Score: 319.70, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 533, Score: 294.81, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 534, Score: 278.00, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 535, Score: 508.60, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 536, Score: 489.63, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 537, Score: 471.94, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 538, Score: 487.25, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 539, Score: 480.97, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 540, Score: 465.51, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "*** Episode 540 \t Average Score (over agents): 465.51 \t Average Score on 100 Episode: 454.63, Time: 02:03:15***\n",
      "Episode: 541, Score: 499.91, Max: 508.95, Min: 47.15, T-max: 999   \n",
      "Episode: 542, Score: 510.83, Max: 510.83, Min: 47.15, T-max: 999   \n",
      "Episode: 543, Score: 508.17, Max: 510.83, Min: 47.15, T-max: 999   \n",
      "Episode: 544, Score: 471.01, Max: 510.83, Min: 47.15, T-max: 999   \n",
      "Episode: 545, Score: 501.54, Max: 510.83, Min: 47.15, T-max: 999   \n",
      "Episode: 546, Score: 496.54, Max: 510.83, Min: 47.15, T-max: 999   \n",
      "Episode: 547, Score: 512.52, Max: 512.52, Min: 47.15, T-max: 999   \n",
      "Episode: 548, Score: 495.51, Max: 512.52, Min: 47.15, T-max: 999   \n",
      "Episode: 549, Score: 498.56, Max: 512.52, Min: 47.15, T-max: 999   \n",
      "Episode: 550, Score: 500.91, Max: 512.52, Min: 47.15, T-max: 999   \n",
      "*** Episode 550 \t Average Score (over agents): 500.91 \t Average Score on 100 Episode: 457.66, Time: 02:05:28***\n",
      "Episode: 551, Score: 475.95, Max: 512.52, Min: 47.15, T-max: 999   \n",
      "Episode: 552, Score: 475.33, Max: 512.52, Min: 47.15, T-max: 999   \n",
      "Episode: 553, Score: 516.72, Max: 516.72, Min: 47.15, T-max: 999   \n",
      "Episode: 554, Score: 491.04, Max: 516.72, Min: 47.15, T-max: 999   \n",
      "Episode: 555, Score: 506.06, Max: 516.72, Min: 47.15, T-max: 999   \n",
      "Episode: 556, Score: 494.99, Max: 516.72, Min: 47.15, T-max: 999   \n",
      "Episode: 557, Score: 501.49, Max: 516.72, Min: 47.15, T-max: 999   \n",
      "Episode: 558, Score: 516.74, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 559, Score: 434.31, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 560, Score: 493.53, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "*** Episode 560 \t Average Score (over agents): 493.53 \t Average Score on 100 Episode: 458.25, Time: 02:07:42***\n",
      "Episode: 561, Score: 486.08, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 562, Score: 492.45, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 563, Score: 499.75, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 564, Score: 497.15, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 565, Score: 483.65, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 566, Score: 490.91, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 567, Score: 473.33, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 568, Score: 485.91, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 569, Score: 464.49, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 570, Score: 474.43, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "*** Episode 570 \t Average Score (over agents): 474.43 \t Average Score on 100 Episode: 460.20, Time: 02:09:57***\n",
      "Episode: 571, Score: 496.28, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 572, Score: 474.34, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 573, Score: 486.31, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 574, Score: 493.26, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 575, Score: 503.29, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 576, Score: 487.57, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 577, Score: 310.87, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 578, Score: 298.32, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 579, Score: 317.83, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 580, Score: 491.42, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "*** Episode 580 \t Average Score (over agents): 491.42 \t Average Score on 100 Episode: 458.97, Time: 02:12:10***\n",
      "Episode: 581, Score: 494.42, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 582, Score: 481.54, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 583, Score: 411.31, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 584, Score: 431.23, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 585, Score: 452.54, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 586, Score: 502.60, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 587, Score: 476.13, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 588, Score: 473.83, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 589, Score: 473.54, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 590, Score: 443.69, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "*** Episode 590 \t Average Score (over agents): 443.69 \t Average Score on 100 Episode: 462.60, Time: 02:14:24***\n",
      "Episode: 591, Score: 476.50, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 592, Score: 508.72, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 593, Score: 493.29, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 594, Score: 490.92, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 595, Score: 508.87, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 596, Score: 487.83, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 597, Score: 485.45, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 598, Score: 507.17, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 599, Score: 504.26, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 600, Score: 494.30, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "*** Episode 600 \t Average Score (over agents): 494.30 \t Average Score on 100 Episode: 468.00, Time: 02:16:46***\n",
      "Episode: 601, Score: 495.34, Max: 516.74, Min: 47.15, T-max: 999   \n",
      "Episode: 602, Score: 517.14, Max: 517.14, Min: 47.15, T-max: 999   \n",
      "Episode: 603, Score: 507.90, Max: 517.14, Min: 47.15, T-max: 999   \n",
      "Episode: 604, Score: 501.56, Max: 517.14, Min: 47.15, T-max: 999   \n",
      "Episode: 605, Score: 490.26, Max: 517.14, Min: 47.15, T-max: 999   \n",
      "Episode: 606, Score: 520.44, Max: 520.44, Min: 47.15, T-max: 999   \n",
      "Episode: 607, Score: 522.35, Max: 522.35, Min: 47.15, T-max: 999   \n",
      "Episode: 608, Score: 511.46, Max: 522.35, Min: 47.15, T-max: 999   \n",
      "Episode: 609, Score: 513.87, Max: 522.35, Min: 47.15, T-max: 999   \n",
      "Episode: 610, Score: 521.52, Max: 522.35, Min: 47.15, T-max: 999   \n",
      "*** Episode 610 \t Average Score (over agents): 521.52 \t Average Score on 100 Episode: 472.62, Time: 02:19:07***\n",
      "Episode: 611, Score: 510.91, Max: 522.35, Min: 47.15, T-max: 999   \n",
      "Episode: 612, Score: 522.97, Max: 522.97, Min: 47.15, T-max: 999   \n",
      "Episode: 613, Score: 88.97, Max: 522.97, Min: 47.15, T-max: 999   \n",
      "Episode: 614, Score: 50.10, Max: 522.97, Min: 47.15, T-max: 999   \n",
      "Episode: 615, Score: 59.90, Max: 522.97, Min: 47.15, T-max: 999   \n",
      "Episode: 616, Score: 543.06, Max: 543.06, Min: 47.15, T-max: 999   \n",
      "Episode: 617, Score: 566.96, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 618, Score: 558.75, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 619, Score: 505.82, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 620, Score: 494.08, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "*** Episode 620 \t Average Score (over agents): 494.08 \t Average Score on 100 Episode: 467.53, Time: 02:21:33***\n",
      "Episode: 621, Score: 500.72, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 622, Score: 394.93, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 623, Score: 412.00, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 624, Score: 417.56, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 625, Score: 500.46, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 626, Score: 499.38, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 627, Score: 547.69, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 628, Score: 509.79, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 629, Score: 501.27, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 630, Score: 506.26, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "*** Episode 630 \t Average Score (over agents): 506.26 \t Average Score on 100 Episode: 468.06, Time: 02:23:58***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 631, Score: 539.35, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 632, Score: 522.55, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 633, Score: 534.39, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 634, Score: 506.80, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 635, Score: 520.52, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 636, Score: 544.43, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 637, Score: 532.99, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 638, Score: 531.60, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 639, Score: 542.19, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 640, Score: 547.14, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "*** Episode 640 \t Average Score (over agents): 547.14 \t Average Score on 100 Episode: 478.23, Time: 02:26:30***\n",
      "Episode: 641, Score: 541.97, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 642, Score: 539.07, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 643, Score: 503.59, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 644, Score: 515.42, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 645, Score: 479.67, Max: 566.96, Min: 47.15, T-max: 999   \n",
      "Episode: 646, Score: 568.30, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 647, Score: 557.69, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 648, Score: 563.87, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 649, Score: 553.27, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 650, Score: 535.90, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "*** Episode 650 \t Average Score (over agents): 535.90 \t Average Score on 100 Episode: 481.86, Time: 02:28:44***\n",
      "Episode: 651, Score: 556.40, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 652, Score: 530.94, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 653, Score: 550.90, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 654, Score: 558.90, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 655, Score: 562.24, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 656, Score: 561.26, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 657, Score: 557.27, Max: 568.30, Min: 47.15, T-max: 999   \n",
      "Episode: 658, Score: 573.10, Max: 573.10, Min: 47.15, T-max: 999   \n",
      "Episode: 659, Score: 571.72, Max: 573.10, Min: 47.15, T-max: 999   \n",
      "Episode: 660, Score: 563.63, Max: 573.10, Min: 47.15, T-max: 999   \n",
      "*** Episode 660 \t Average Score (over agents): 563.63 \t Average Score on 100 Episode: 488.66, Time: 02:31:02***\n",
      "Episode: 661, Score: 542.53, Max: 573.10, Min: 47.15, T-max: 999   \n",
      "Episode: 662, Score: 549.02, Max: 573.10, Min: 47.15, T-max: 999   \n",
      "Episode: 663, Score: 537.04, Max: 573.10, Min: 47.15, T-max: 999   \n",
      "Episode: 664, Score: 572.44, Max: 573.10, Min: 47.15, T-max: 999   \n",
      "Episode: 665, Score: 573.35, Max: 573.35, Min: 47.15, T-max: 999   \n",
      "Episode: 666, Score: 553.70, Max: 573.35, Min: 47.15, T-max: 999   \n",
      "Episode: 667, Score: 550.47, Max: 573.35, Min: 47.15, T-max: 999   \n",
      "Episode: 668, Score: 558.00, Max: 573.35, Min: 47.15, T-max: 999   \n",
      "Episode: 669, Score: 581.93, Max: 581.93, Min: 47.15, T-max: 999   \n",
      "Episode: 670, Score: 556.37, Max: 581.93, Min: 47.15, T-max: 999   \n",
      "*** Episode 670 \t Average Score (over agents): 556.37 \t Average Score on 100 Episode: 495.93, Time: 02:33:18***\n",
      "Episode: 671, Score: 583.61, Max: 583.61, Min: 47.15, T-max: 999   \n",
      "Episode: 672, Score: 585.86, Max: 585.86, Min: 47.15, T-max: 999   \n",
      "Episode: 673, Score: 545.48, Max: 585.86, Min: 47.15, T-max: 999   \n",
      "Episode: 674, Score: 569.44, Max: 585.86, Min: 47.15, T-max: 999   \n",
      "Episode: 675, Score: 546.35, Max: 585.86, Min: 47.15, T-max: 999   \n",
      "Episode: 676, Score: 567.17, Max: 585.86, Min: 47.15, T-max: 999   \n",
      "\n",
      " End of PPO!\n"
     ]
    }
   ],
   "source": [
    "# def ppo(env, agent, n_episodes=2000, max_t=1500, print_every=100, n_agent = 12):\n",
    "def ppo(env, agent, n_episodes=1000, max_t=1000, print_every=10, n_agent = 12):\n",
    "    \"\"\"\n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        print_every (int): number of episodes to print result\n",
    "        n_agent (int): number of identical agents in environment\n",
    "    \"\"\"\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_global = []    \n",
    "    scores = []\n",
    "        \n",
    "    time_start = time.time()\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations\n",
    "        agent_scores = np.zeros(n_agent)\n",
    "        t_max = 0\n",
    "        for t in range(max_t):\n",
    "            actions, log_probs, _, values = agent.act(states)\n",
    "            # get needed information from environment\n",
    "            env_info = env.step(actions)[brain_name]\n",
    "            next_states = env_info.vector_observations\n",
    "            rewards = env_info.rewards\n",
    "            dones = np.array([1 if t else 0 for t in env_info.local_done])\n",
    "            agent.save_step([states, values.detach(), actions, log_probs.detach(), rewards, 1 - dones])\n",
    "            states = next_states\n",
    "            agent_scores += rewards\n",
    "            t_max = t\n",
    "            if all(dones) or t == max_t-1:\n",
    "                agent.step(next_states)\n",
    "                break\n",
    "                \n",
    "        score = np.mean(agent_scores)\n",
    "        scores_deque.append(score)\n",
    "        scores.append(score)\n",
    "\n",
    "        print('Episode: {}, Score: {:.2f}, Max: {:.2f}, Min: {:.2f}, T-max: {}   '\\\n",
    "              .format(i_episode, score, np.max(scores), np.min(scores), t_max))\n",
    "        \n",
    "        if i_episode % print_every == 0:\n",
    "            torch.save(agent.actor_critic.actor.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.actor_critic.critic.state_dict(), 'checkpoint_critic.pth')\n",
    "            s = (int)(time.time() - time_start) \n",
    "            print('*** Episode {} \\t Average Score (over agents): {:.2f} \\t Average Score on 100 Episode: {:.2f}, Time: {:02}:{:02}:{:02}***'\\\n",
    "                  .format(i_episode, score, np.mean(scores_deque), s//3600, s%3600//60, s%60))\n",
    "            \n",
    "        if len(scores_deque) == 100 and np.mean(scores_deque) >= 500:   \n",
    "            break\n",
    "\n",
    "\n",
    "    print('\\n End of PPO!')\n",
    "    return scores\n",
    "\n",
    "scores = ppo(env=env, agent=agent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXecXGW5+L/PzPaatullExICIZSQ0IsQernCFRDwKqhIRLBdBQ0iKor3gj8QQbwgihqkCALSAgiGUIWQBJKQQgohveymbrLJtpn398c5Z+bMzJnZ2d2ZnfZ8P5989sx7yj6zmXmf96mvGGNQFEVRlHj4Mi2AoiiKkt2oolAURVESoopCURRFSYgqCkVRFCUhqigURVGUhKiiUBRFURKiikJRFEVJiCoKRVEUJSGqKBRFUZSEFGVagJ4wYMAAU19fn2kxFEVRcor58+dvM8bUJXt9WhWFiPQB/ghMBAzwVWA58DhQD6wBPm+M2SkiAtwNnAvsA75sjPkg0fPr6+uZN29e2uRXFEXJR0RkbVeuT7fr6W7gZWPMQcDhwDJgOjDLGDMOmGW/BjgHGGf/mwbcl2bZFEVRlCRIm6IQkRrgZOBBAGNMmzFmF3ABMMO+bAZwoX18AfCQsXgP6CMiQ9Iln6IoipIc6bQoxgCNwJ9F5EMR+aOIVAKDjDGbAeyfA+3rhwHrXfdvsMcURVGUDJJORVEEHAncZ4yZBDQTdjN5IR5jMT3QRWSaiMwTkXmNjY2pkVRRFEWJSzoVxQZggzFmjv36SSzFsdVxKdk/G1zXj3DdPxzYFP1QY8wDxpgpxpgpdXVJB+0VRVGUbpI2RWGM2QKsF5Hx9tBpwFLgOeBKe+xK4Fn7+DngCrE4FtjtuKgURVGUzJHuOopvAY+ISAmwGvgKlnJ6QkSuAtYBl9jXvoiVGrsKKz32K2mWTVEURUmCtCoKY8wCYIrHqdM8rjXAdemUR1EUJZeZtWwr9QMqOaCuqld/r7bwUBRFyQFWN+7lqhnz+N7jC3r9d6uiUBRFyXI+XLeTqXe+AXikgvYCqigURVGynNc+bggdlxX7e/33q6JQFEXJIfziVXKWXnK6e6yiKEq+sbO5jW//7UOuP3M8laV+vv/EQkqLwlZEBvSEKgpFUZRsYunmJt5auY1/f7Kdq08aw8INuyPO//uT7TTsaWFgdVmvyaSuJ0VRlBQwf+0O3ljRtbZC33t8AfXTZ0aM7dzXBkAgaGgPBD3vO/qXs3h75bbuCdoNVFEoiqKkgIvue5cr//R+l+55+sONACzd1BQa29ncFjpu2NMa995Nu/Z3UcLuo4pCURQlhbR1eFsBiTj3nrcAePGjzdz87JLQ+Mqte+LeU13We5EDVRSKoigpZN2OfUlfW+yPjEzf+sLSiNeNCSyK6rLirgnWA1RRKIqipJCtTS1JX1tXVRo6/sbD89m0O/Le7S43VDRqUSiKouQo8QLQXkwYWhs6fmnxFgBuOvdg7rzk8E7vVUWhKIqSo7QHkm+yETSx1549cTBTDxoYM3750SMYU1cZeq2uJ0VRlByitSMQOnYsCmMMT87fEHEOIBg0LNlk1UZ0BA2lRZHT8JDaMqpc1sLwvuXUVZfy+SkjIqwVtSgURVFyiObWWEXx6tKtXP/3hfzmXytD5z7e0sQf317Nefe8zfy1OwgGDQNccQqAIr+PYn94ai7yCXNvOp1JI/vS4bJWerPnk1ZmK4qi9JDm1o7QsZMe29RijW21A9SrG/dy9m/eCl03f+1OOoJB+lWWsDFBTcRe17Pv++Jkfv78Er41dVxK5e8MVRSKoig9pM3lEnJiFD4787W5rYPXPt5K34qSiHv+tawBDFSWelsGJ40bwFsrt7FrX3to7IgRfXj62hNSLH3nqOtJURSlh7hj0o7ryWne988lW/nqX+axKKpn0/uf7mB7cytFPu9p+A9XWJuDdgQzsQNFJKooFEVRUoijKHxRbV5veT5ccV3fvwKAPS0d+H3e7WDLiv1MGdWXO5JIlU03qigURVF6THjV3xayKCIVgNswqB9gpbk2t3ZQ5BPemT7V86lPfuN4Lp48PMWydh1VFIqiKD0kwvXUYewxb5dR/8oSjj+gPwDNbQF8PmFYn3IATj84tn4iG9BgtqIoSgpxXE/xmgPOvuEU5q/dGXpdZLuelt96dtx4RabJTqkURVGynK1NLRhjmL92R0TzPkdRxKvQriwposy1Y53PVhSlRf648YpMoxaFoihKF/n7vPXc8OQihtSWsTmqkV9byKIIeN2K3yeUFkcW1GU7alEoipJ3BIMm5PrZ3+Y9YXeHlvYAn25r5rH31wHEKAkgVD2dqOeT26LwZ2IT7C6iikJRlLzju48v4MAfv8RLH23m4J+8zLLNTZ3flAQ/eXYxp97xOolKG0IxigRdZMtcFkW2upvcqKJQFCUvaGkP8LvZq2jtCPDcwk0APL/I+rmqYW9KfsfcNVYQenVj5PPcE3+bRzC7trw46nqXRZEDikJjFIqi5AytHQG27G5haJ9yAkFDWbGfZz7cyG/+tYILjhjG3bNWUlESnoQbmqwgc/REnSy77fYZbYEgddWl9KmwnuP0cXLoU17ClnbLDeW4nNwWRf/KEnbvD7ficCuKQBZUXndGWhWFiKwB9gABoMMYM0VE+gGPA/XAGuDzxpidYlWn3A2cC+wDvmyM+SCd8imKkl3sa+ugrMgfygSK5puPfsirS7cydmAVqxr2sua287h39irWbN/HE/PW288IxyQcl1N0K+9kOfznr4SOrzv1AEpcXV2njOrL7Rcfxml3vsHoAZVssXe2a7ctiXaXRdG3sgS2NYfud1sg2dCiozN6w/V0qjHmCGPMFPv1dGCWMWYcMMt+DXAOMM7+Nw24rxdkUxQlS2hpDzDhJ//k9n9+HPeaV5duBSJdSf3sZntOYNkdG262lUYgTvFbS3sgYqXvJnql/7vZnzDn0x2h1yP7V3BAXRXVZUURGxC12NlOboticE0ZADf/xwTASoV16MqOeJkiEzGKC4AZ9vEM4ELX+EPG4j2gj4gMyYB8iqJkgD22O+ev767t0n3bm1sjXq/cGhuPiOfeueDedzj8llcixm5+ZjGXP/AeLe2Js6XKXe4j99OXbmpib2tHRG3F8H7lrLntPL507CggMi7R0YUd8TJFuhWFAV4RkfkiMs0eG2SM2Qxg/3Rq1ocB6133brDHFEUpAJw01n1dSGdt7QiE4hAO//hwY8x1Le1BPn//uyGLxGH51j0x1/71vbW8u3p7p4rCiTMIhDRFv8oSGva0ctH//Tu0BzbA5ybF9mu67XOHAmpRAJxgjDkSy610nYicnOBaL6dkjKoVkWkiMk9E5jU2NqZKTkVRPHjtY2uXtnh9i1JJc1tHwvMdHhPq7n3t7GlNfB/A8i1NvL9mB1c/NA+ADTv3cf8bn4TOe7Xb2N+JonDOi0jI9eRYCo4CGtW/gjW3ncf4wdUx9/e3d7ZrL/QYhTFmk/2zAfgHcDSw1XEp2T8b7Ms3ACNctw8HNnk88wFjzBRjzJS6urp0iq8oBc/Nzyzhyfkb+GDdzs4v7iEL1u9KeP6VKGsAoHFvq8eVsdzxyorQsTGGqx+az20vhWMhO5rb+Oy9b/P43HWhsXgWRXWplQPkxDZEwivaaH163alj48rkBNi9FGC2kTZFISKVIlLtHANnAouB54Ar7cuuBJ61j58DrhCLY4HdjotKUZTMcMDAKgA+XLeLYJpXvjc+/VHC8w1NsVXQW6PGnrzmOA4f0Sfhcx6es47d+9oixmZ+tJlFG3bzs+eWhsa8XGDTzzmI/73Ichntdu0851hc3z/zQAAOHVYLJK66LrEVRaG7ngYBb4vIQuB9YKYx5mXgNuAMEVkJnGG/BngRWA2sAv4AXJtG2RRFSQK/Pc/dOnMZp9zxOut37OOp+RtS/nsS1RK0B4L8+pXlbHHFIpz599WlDRHX1pYX07ciXDPx+y9Njnnezc8sjvFpf7TBsmYG15aFxnbui82GuuYzB3DMaKtF+OeOtEKoQnivibqqUkYPqMTYv6HIH19RFPsdRZH9rqe01VEYY1YDMVszGWO2A6d5jBvgunTJoyhK13Gvqtft2MdJv5oNwOkTBnW7iM2LvQniDM8v3MQ9r62KGKsqLWJPS0eo55JDeYmfSts1dPHk4YwfFBsbgNgeTbtsN9J2lyvLOb7/i0dyzcPhkq666lLW3HZe6LWIRCgen4RjHomqrottJdIRLGyLQlGUHCfeBL52e3Po+OXFW7juke7Vxra0B5j+1CLWbd8X95qVHu03asq8lVR5sZ9KuzK7tMgXMVH//IJD4v4Ox6JxV1zvaLbcU8P7VnDDWeO53nYreWK7nkQs5eAoikSdYR2LQtNjFUXJCfa2dtDSHmBVw17qp89k3podtLQHWLLJu5nep9vCiuKah+cz86PuhROfW7CJv81dz83PLvY839TSzn2vfxIzXlnq97g60qIoLfJHuH7+65hRcS0ML9fXdltRlBX7ue7UsXxz6jjPe92uJxFrr+ywRRF/inUsikTNA7MF7fWkKAoTf/rPiNcPvLk6lL7phZcFEAiaLje4c2INzuo9ml3N3lXT8Sgr8lNZYk1rxUUSIY/fJ1wyZTi3zlwWc5+norBdT+Ul3krJwcp6Ct/v9wltASddNv59w/pUUFLk4/ozxyd8fjagikJRlBi8UlEH1ZRy9Oj+vP5xQ6ivkbu+oq0j2OmkGo3jflm3w1I8Bw2u5uMtezDGICJxayvixb59PqGu2lJwJX5fTNZRvNW7l6J42S6Yc1dgeyM4YQZBbNeTFdtJZFGUl/hZces5nTw7O1BFoSgFTmucndgcfnPpETTsaeFLx9ZTXuLnnLvfYt2OfTw6Zx0/+kc4pbU7iiI6K8hRFIGgoXFvC/fOXuV5X6JU3UuPGoExhjMOGRyzB/W4gZGup2+eOpZ7Z6/ybMznxCvcDfziEbrbcT0FOo9R5BIao1CUAiUQNPz6leUs3xLbxsLhxnMO4sJJw5h28gEhJTC4ppS3Vm6LUBIAX/7L+2zctb9LMkRP5H3sBn8dQcO1j3zAzEXesY/oiX2oK621rNjPl08YzbA+5fijFNEZEwZx6nirUPeGs8Zz0WSrtUYwqlJuTF1l+HlFSbieTJTrKYmsp1xCFYWiFCDBoOGAH73IPa+t4luPfRj3uol24Zgbd62Bmw/X7eLe17wtAAdjDG+uaAxNrNET9BD72YGgidjCNHplHjSGY0b3CymIR68+1tON47WiH1ht3VNTXoxzOtr1NMAVn4nX8txBCFdkC1aRnfM4VRSKouQk1z3yAT97fkno9fa93oFkgLF2ZbabQTXeigKgX2Xi2oo/v7OGK/70figGEl2VHEoZDZoIN5YzcTvxh2DQ4BPhC8eMBKBvRUmo0tmN10TtZEUNrS3DZ8cwohVFV2pE3MFsEcFtJOWLotAYhaIUGNGprImK3QZ4ZD4NTqAo+tquo3i8s2obYE3MwaCJqUp2YhYdgWBEEHlgTSlbmloY0becxj2tdAQNIlYvpa+eOJqKEu+pzKuFxvfPPJCxA6uYetDAkKss2pXV1WJCt2HkVg4ao1AUJeforHU2WH58B68VcaK02aAx/G72Kv79yTbP8079RVmxjwk/fZnr/74wdO7xaceGfl8gaCIUxSFDa4BwqmrQWBaFiMRVEuDtNqosLeILx4xE7Pud3+emSxYF4e6xAiErBdSiUBQlBznx9tmdXjOwOr4iAKgpi5w2Jg6rYfFGqzCvpT3Ir1+1OrW621w4ON1ed+9vp6U90u10zJj+rLErvqNdT9PPOZiq0iLG1FXxzqrtBGyLoqfEi1HEq/z2wt091qnMdogO1ucq+fEuFEUB4LH31zF7ebhR3tsrt/G+a/vObXHacvdxNdILBA2Da8qYNNK7C+vRo/vxiwsO4ZyJgwE4c8Lg0LlEFssvZy4N7WK3M04hnVN3EG1R1JYXc9N5E6iwlUeiJoJdIX6MootraLfrKcKi6LZoWYVaFIrSDZ5dsJF9bQEuP3pkRuX4dFszzy7YyHdOG4eIhFp1f/yLsykr9vPFB+cA3qt7N/defiRNLe1c+8gHNOxp5b0fxfTtDCEifOm4ei6aPJyTxtUxfnB1yIrw2uzn168sZ9ygav7w1qehsZ1Rbb7n2L/P8el3BE0oOO12hTmuoqCJdPF0F4ljUdRWdMX1hMv1JBHurkQFd7lEfrwLRellvvO3BZ3un9AbfOnBOfzmXytDfYkcnJ3ckuW4A/ozrE85ELvHQzwqSixff6kr26jZIzDulYL726g02v6VVhA8HKMI0hEwlPh9/N9/HRm6zu0qSo3rKayY3HTN9SQRvZ7cFoUGsxVFiUnv/Pcn2zjx9tfY02K5Vv741mqe/iD1+zc4NO5p9ZTjrZWRweQ9Le3848NYOf7ylaOY/+PT8fskVGR2yeTY/Z0T4VYUT8yL/B3JuogcBeFMrO0BQ3sgyMCa0lDKLLhcRcZ47p3cVZxnBKJafVd3QVFAbK8nr+NcRhWFovSAcTe9xIad4QZ5d76ygg079/Phul0EgoZbZy7je08sTPCEntFqVwDv99iNzc2rS7fy349bcnzx2LC7rH9laSiLqbqsmE//91y+fMLoLsngVb/g4GwX2hmOS6nIH45RtAdNhJKA8MTu1FH0lHgxiho7RnFknDhNNO6CO18eKgqNUShKFPPW7ODi+9/l1f8+mXEebambWiInv8Ubd3P93xcysl8F/WwXytMfbKCs02ZysHHXfva3dTB2oHf762SJziACeHTOOvw+IRA0POWyatxB4ug+RtKNyVfirO2DQRO3K2w83DGKjkAw1Io7Wr6ASa3rKdrwqS4r5uNfnJ3URG+18HBehHcFhPxRFGpRKEoUzyzYCMC7q7cD1mqzw+Xa+dqMSP//hp37eW/1Dp6YtyE0YTyzYBOf//27AFSXxV+PnXDba5z+6zd7LPP+9kCEjAA/+sdHlNgr8rlrdobG3YqitJM+RslQVuI9jby9aluEoqgpK+K8w4YkfJY7RtEeCMaklzrzrvV3TkEw23589C5z1WVFlBX7Yywaz2dE9XryacGdouQ/zo5jziT1H799m7E3vWSfC0akm0LkdqFuN5SDOzC6bHMTizfuTrnMLe2BkBsKwpviOFlIba5zZa76hNIkOqN2xsDqMl7575MZUBVZlf3we2u59pH5oddNLR2M7FcRen2+h9IIWRQBq2o7nkUBYaXRE8IxikiToipBEV/sM8JboQoSlR6rikJRcpJ/LtnCwvW74p4PtYi2J6mlm8O7vDl7K5920MDQmNsP/7FHJ1Z3i4xz7n6L83/7dsw1xvSsLsCtKG757CH8+LwJca8t8ftCcYXSBPGFrnDgoGre/uHUUOYUWHtabIvqI+WueL73C0cSjbsyuyMYDMUsHNzzbrKup/u/OJlnrjvB81y8GEVnjQCjid/CIz+m2Px4F4qSJIGg4et/nc8Fv3sn7jWORSFY8Qo3zsp8YE24enl7nCI2h31tHfzq5Y+pnz4z7jVe9QddYX97IOTmKS3yRRTQRdO0vz0UpE2F68mhrNgfcrONGVDpec3xB/RP+AxHObcHDe0dsRaFO4AdLzYSzdkTB3PECO+gdLwYRVcQcdVRSKSSyRM9ocFspbBYttl7D2g3jr+6cW8rNzy5KOKcoyjqqsON8aJrGKJpDxj+z2PfZzc797Un7FnkEAgavvfEAr5ywuiIyW/Ntma++ahVq1Ba7Avt6+BF4942HrhiCss2NXV5o6HOcCbemji9kg4bHjlh/+t7n4kovgtXZgdpDwapKo76m7h0Qyom4VQExAVXCw+i6yjyQ1Pkx7tQlCRxegkloq3D+trPcwWAHRz3jrsfUnSb7v938WExK+Fo6qfPDNVaAOzal1x2UOOeVp5dsIlpD82LcJesce1hXVrkD7W6cDOmrpLhfcu58vhR1JQVc8yYxKv77uC4XZJtqjd2YBVH1fcLvXbHKDoCJiYY3B2LIhGpUBQQ3/WUJyEKVRRKYbHTXv0n+gI7FsVrHzfEnHMsCnfgdk9rZLrspJF9OXti4uwegNte+jh0vHtfcvUGzu9v2NPKT55dHBp3V0SXFvkY3jccK3Am27F1Vbz9w6kcNLgmqd/VHZy/azyLojMc19OWphY+2rg7xiXUnRhFIlLTBkRCMSYRCT2zyCfdSjfORlRRKAXFDlczuhufXkT99JkYY9jZ3Mbf563nvHveYuXWvZ73nnP3W/zyxaUAlLvcRHvtRncXHjGUH593MGMHVoU6rNYl6MT6yJx1oeOvzpjLmXe9wfodsVlTbprbwgrBfb/TbA+sGoQhteVcedwoIDwZJopbpArHP1/uyqa64IihSd/vKDWnd9QbKxojn++2KFJYcNcTIno9SbgRYFcD4tmMKgolb/l0WzMn/2o2W3aHexc5/vCggcfeXw9YlsOkX7zKDU8uYsmmprj7Pi/b3MR7q63gtjtbyJmkv3RcPV87aQwQdr24V/aJaGkPsmLrXq595IOE13n1UgIrYO7QZGdhOXGKgD2JJRMD6SmOf94Jkh9d348fnH1Q0vc7dQvxfPsS57i7uOfybqeySkTz2JCCGFSTuF17LqGKQslLWtoDnHrH66zbsY9/LtkSGveqFE6UKhuPkiJfqJGd01DOHZdwFIVzTbI4imDz7v38z4vLaGkP8EnjXuqnz2Txxt00x2nVsd9VmX3iuAEAfOGYkRw8pIbffWESpx88kGknj+mSLN3BWaG723p0pSZhaJ9yyop9cduhp7yOwvU85/+vO0Vy0XtmAxw5sm+P5csWVFEoeckcV1FcXXUps5ZtpWFPS0x7a4CtTfHTW+NZBCV+H/NvPoOT7EkZIlfBFfa+zF3pQgpQVVZEU0s7f35nDQ+8uZo7X1nOMx9aleLPfLiRu/+1wvO+/bZF8fS1xzOk1pJ5UE0ZL33nJM6eOIQ/XnkUQ/skZ930hB+ecxCj+lcwZVR4kqwoTT6zqtjvY+LQ2tDrR752TMT5yBhFalw7zmOcivXrzxrftfuJdD05SQbjPPYbz1XSbouKiB+YB2w0xpwvIqOBvwH9gA+ALxlj2kSkFHgImAxsBy41xqxJt3xKfuLuBho0hqtmzOOgwdWeBWZrd0RmQp176GAuP3okfStK+OXMZWzYGeuKcnokufs5uS0K5/eICNWlRexJsC+1m027Wph6xxuhFfWMd9eGNghqbgvwwbqw9VPkk5A149RhZLplxORRfXnjhlOZY7c/gbA7yVGqL33npIg2ItF8/qgRzFtrZZydMHZAxDmJyHpKDT4RAsZQ5Pd1um+HFyKRvqcm2xVZ28n+4blEb9RRfAdYBjipFrcDdxlj/iYi9wNXAffZP3caY8aKyGX2dZf2gnxKHuJOWXWO1+/Yx+DacP3DoJpStja1snB9uKXGeYcO4Xeu/Q+im+Y5lPj99vnwhOeuInYURWtHgJry4qQUxeCaMrbuaYlItWzrCLJ5lxVjeX7hpojra8uLQzUcThuRbGkZEb3af+/G00LB9IOHJM66ilYObtJhUfgEAnTfleW2KEBCTSOjt4zNZdLqehKR4cB5wB/t1wJMBZ60L5kBXGgfX2C/xj5/muRLblkeEQwa7nxlOZviBHyzBXcRnHNcWuynuTUQmvz7VVrBRndVdE3UFpjxJl7HB++2XNyreUdRtHUEk26TMai2DK9OHo67zN0KZNHPzowolmsJWRTZ4U2ObkkyuLYsqW66AP0SrMQjLIoUzQ7OM3uSAeXeM9tJbuhuinA2ku5P1W+AHwDOt6k/sMsY43ziNwDD7ONhwHoA+/xu+/oIRGSaiMwTkXmNjY3Rp5U089HG3fz2tVV8P417LPQEZ8J0t9XY0Wwdl/h9NLd2hALNbR2xgeHomEK8ycNRFO4+T+5Oo0eP7k9FiZ9pJ48h2e4QQ2rKPMdXNsSm61aXFkUU1bXbbUeyxaJw4iEnHxjfOohHompx939Hqt6p85zuKoqINuOEs87UokgCETkfaDDGzHcPe1xqkjgXHjDmAWPMFGPMlLq6uhRIqnQFJyunLRDkiXnrU7bJfSp4efFmDrr5ZZZuauLlJVtCNQyO62lLUwt7WjtCysCZXN1E1z14VThD2GJwK4oiV4yiX2UJS39+NlPq+yXd8M/tFkvEBzefgYh4prtmOkbhMKJfBe/eOJVrTxmb0uf6IrKeUuV6sp7T3ccJ4tozm1DFfVcTGbKZdFoUJwCfFZE1WMHrqVgWRh8RcT7hwwHH8boBGAFgn68FIjuyKRnH8YXPX7uTHzy5iBn/XtOt57R2BAimQMkYY/jtrJWsatjDrGVWJfXs5Q2s37Gfy4+2dnJ7afGWiHsciyJ6/waInazjVVh7WhRx3D7R7/LBK6d4XpdMzcW3p44NbY7kpcSyxaIAGFJbnvKis1RXZrufmRrXk/Dl4+sBGJZkDU0ukDZFYYy50Rgz3BhTD1wGvGaM+S9gNnCxfdmVwLP28XP2a+zzr5me9l5WUs6+qC6n0ZWzyTL+xy9z/ZNh91VHINitVtvNbQHufHUFl9z/bmhvBcf0j1cV7az82wImZnIeEqUozjpkEH+4InZid1buZxw8ODRWXOQ90QRd76u6rIjjXB1US/w+jhjRh+qyIg53Nfn74dkH8derjo551kCXe+qr9palo12dWos66TGVKzz1jeN44Vsnxoy7+zulTlH00KJwdwXEKrxcc9t5vVLg2FtkIvL1Q+B7IrIKKwbxoD3+INDfHv8eMD0DshUMqxr2RgRHk+XROWsBqLRXsyu3xu6/0BmOu+rpDzaGxi743TucePts2j1W+YlwXGE797WHsoOcVX5lHLeRs790eyAYWp079I0KpIoIZ0wYxP1fDGdCHTemfygA+qNzw1XH8QLJjp547OpjWfTTM0O7zgGs+OU5PPH145j349MjdsIb3rc8RjaIXPWePmEQa247L2IPiGyyKHrC5FH9mDisNmY8IkaR4jqKnlgUbtdTPtIrKs8Y8zrwun28GohZKhljWoBLekOeQsQYQ1NLBz96+iN+fsEhnP7rNziqvi9/v+b4pJ+xp6U91MKiuMgHbQG2NLV0clckc9cHwdKUAAAgAElEQVTs4JL7340YCwQNSzZZ7b/nr93JsV3oaupWdrPsJn4Ne6zgdbwsm1J7vK0jiIhwxyWHs2tfG/vaAhGrczdnTxzCUfV9mbtmJ9ecckBo3J0SG69jrKMoqkqLEJGYVb/jxqoqDX8dh/Yp96w18LIY3FXQ2ZL1lC4iu8emBumxRSFJJyzkKvn9qVIAuGfWSkbf+CJ/e38dMz/azF12de9cjzba0Ty/cFOo7cVaVytrJ8MnaCwldMvzS/jGw/Nj7p+9vIH66TNDz3jgzdUx17gzlGYvD3dsnbtmBw22ItrR3Mbufe0xcQ2v3kdO6q575e6eBL5hT/TtgSB+gYsnD+drJ43h26eNS7hKbbOD31VxKo3j3eu41DqrUHZbMyP6lUe4Lg4ZWsOlU0bw2cNjG+y5FVS+uJ7i4f4Tpy6Y3fPnGVdldj6SP040JS6/f8PaNGeHnY+/bU9yex/sbG7jW499yOHDa3n2myeyztXZ1O0i2rnPajkRjTGGP9iK4d1PtnPzs4s9ey25rZLXP27kxnMOBuCS+9+lrrqUuTedzpG/eDV0zcNXHRPqZ+TlPnOa+rn3gz5wYDXLt+7hzAmDqLQn4I6g6ZKrxgl+d3VXOEe1uRXXpVNG8JnxkVl7lS6Loq6qlKb94ffWp6KY2y8+zPP5JS55siXrKV1Edo9N7TN7VnDnHOfn318tigLA6TvkpIm+vGRLossByx308HtWPGLhBqty+WPX7nAtrqB2i8c2nsu37GH0jS8y327FsGD9Tk8lsXLrnlB31/84fCjLt+5h0679IUXUuKeVm59ZHHHPFx+cEzpubo393U7Bk3tCH2QHqUuKfBHKoSt+7vaovbQd4sVCHBwrwN3m+/aLD+PcQ2Mzqu685HCuOnE0IhJRT5AoDdltUeRLjCIeEVlPKXpmyPXUzSfmqxXhRhVFAeBMZOu2J97rwM0LizZx56uRDejeWNHI5FF96V9ZEtrpDTyKXYCnP9gAhHeEc+IG0Zxx15vM/GgzVaVFfO1EK4vn+YWbIvZX+KutsNw4E2e8ttsQ6bt36h5Ki/wRk6m/C9/yC46wakMHRxXGvfq9z/BoVPM6Nz84+yAW/uRMqpPIq79o8nBuPn9CSP7vn3EgkHhP59ICilGkM5jdo6wnj+N8Ir8/VQVIU0s7ry+P3JnNWVmvaPDOULryT+/HrNrdeziA5fdf1bCXw4bXUuSXiEpUr7TW6EZ68TYDApj76Q5OGV/H4SP6cNyY/jz6/rqIbUK9cPZfiHY9nXXIoNCxewJ13D4lRb4I90xX5tVrTzmAZT8/O2Y/6qF9yjk+QX8iv0+o7eamQcfaqbSJak7cFeF5blCkpYVHKmIU+Y4qijzj6hnz+PKf50Zsremki+6Ks93mGysaY1bt0fseHH/bazS3BRhQVRqzao3WEwvW72LmR5sjxtYm2Kt60+6W0Cr9xHEDWLt9H/e/ERv0drO/PcDijbtZ6nKHAdS7spZKi3w8fNUxPHPdCSHrorTI1+3q3mh3UG8wfnA1fp9w3dT4Fc4loc1+8mfrzXikes9s9zO7a4zla1zCjQaz8wxnH4Yd+9pCq1h39XCy7G/zdun0qyyJSQPdF6VULvzdOzH3xdtwx8EpjptgdxZ97P11iS5nf1uA83/7dsz41SeN4fe2kikp8oWC3o/5w4qiKId8+jVlxXzyP+cmvMZJA86nrTfj4X6HqXq74WB2z2MU+aqn1aLIU0JbfgZNRHfU0iIfA6rCrpNT73jd8/7oyd+hX2VJRO0AdE8RRTPQ3jby2DH9OWhwtec1J40bwPdsn707zbZvRTHfP+NArjpxNAOqwtXY7mC2UzVd0gOLIlsJtULP92R+0pP1FIpRdPf+iOPc/zx5oYoij9jvmtx32Ypif1RGUnmJP8J19Ok2b5fQ/kSKImop17AnHM+Ijlf8+LyDGVMXWcT29ZPH8KNzD2LisPC+BINrykPy/Z9rPwg35x82hEOHW9W6j8wJWxy15cV867RxoSCwgzs91nnPJf6oGEVeKAq7gLCLVe25SHqD2bn/WUgXqijygI5AkP99cRnvrt4WGvvqX+axvy1As+1C6mu7ocqL/YxNYovG5jiupyG1ZRHBU4D1O8KBa3dH1onDavjaSWMi6gfAqkCedvIBvPCtk3jhWyfyq4sO46j68NaZ0cFih4qSIio8qpXjfcHdwWzHYtm9vz0y6ykPvgHJ7vOQD6TDzdPTOgrSYOVkG3nwNVHeXrWN37+5mq/+ZV7E+IsfbQ5ZBo5LxifCvV+YxIkeWTqvL29g3pod1E+fyT+XbOWw4bUsueUsrj/TcvccMaIPw/tWxNQRrN8ZTrt1r2o7bKVRErVxj7ux4MRhtXz+qBER7ix3H/8vHTsqdFxa5PMMJnvtgw2RBW5OP6SNu/ZH+PLzwaJItK1ovpHWYHZ3YxTu49z/OHmiiiLHCQQNX/3LXM9zO/e1hQrS+rviEn0qSvjfzx0ac/2X/zyXi119mKpKi6gsLaLcrmQ+ZKjlKopup73RlQrb5IpXHDO6HxCesMcPsmIPxx+QuJeTW2n84sKJ/P2a4xg9oJKDh9R4ttaOzuZ66hvH8fWTx0RYGo7sYwdWRaXH5v43u5Asisj4UmqemYqmgPmOKoocZ/ve1phirLk3nQ5YFdNOvUF/26JwVvcj+lVwzsTBcfeEhvB+B07cwXE5RVsU7jjINrtv01Unjuam86yYgfMFPGviYJbfejYnjevahlNH1fdj9vWnMKJfhWdBWW3UlpOTR/XjxnMPjhgbO7Caf373ZL592ri8C2aXlxTO1zgdrieJOeji/RHV4rn/efIi6fRYETkRGGeM+bOI1AFVxphP0yeakgwdUVqivNhPXXUpRT5hf3sglL1UZysK92p6RL+KhB9sx5JoC/U4chRF5MTkLgZ7ar5VkT15VN+QUmq395WuKStKuk/SU984PhRXceO2jABOHV8XE8SOx3g7m6ooojI7qVuzmrIu9p7KZdIxKaei11PoOA8+T14ktRQRkZ9i7SNxoz1UDDycLqGU5Inus1RTbk3u5cV+fjf7E6740/sA9Lf3NnBP8n6fJOwh5NzT3hFpUZREza4BV6bTjHetwj33Kt+JVXRlI5fJo/oypi426F5dVszPLzgk9PrzU0Z4XpcIX765nnq5CDCTpMP11PM6itz/DHVGsjbrfwKfBZoBjDGbAO9kdyVtbNi5L6KCetayrUy9842Ia5zJPHry6GevxCNaUvsktNr3wmlid8wYK9bgFK9Fu3+8dI1bUTjN9OLtP91VnBV0aZGPsycO7uTqWPIuPdb+e0QnDeQjEf9bWbRxUehZPX5CdpLsEq/NGGNExACIiPfuLkpamfbQfJZubuLsQwZTV13K7z32dojuseRQ39/6L4twu/gkpv2GG2diP3ZMfz7+xdmhoGl0jMKr15NbUThWS6raXzj1EWdMGNSt1Vx3mwJmK06cKTpWk49k5cZFEc/quTzZSLJLkCdE5PdAHxG5GvgX8If0iaV44biZdu+30kEHRXUxhXBn00ZXt9Z3pk8NfYAjXE+uT/U1nwnv2ubgTrt0Z9ZEWwZe7iv3Np5OHCVVFkVP8eeZ68mx8Jz2J/lMOhIRetoUMF+Vg5ukLApjzB0icgbQBIwHfmKMebWT25QUU2OvGHc0t3PmXW+wIqoj67s3To0phvvHtcczrE85ATtOcNYhYVeN32UZ1JYX45NIN9KBg7y9izVR7bKjFcWjXzsmYhOejmBqXU+OAdNd37A/DX7uTDKyfwV3X3YEp4wfmGlR0k8aC+66b1FENvHIRzpVFCLiB/5pjDkdUOWQQRzXwra9rTFK4tYLJzKktjzmHsfqGNm/gg9uPiMik8jthir2CzXlxaGahBPG9ueYOHtX10S5OKI9TwOqSyNeO0oqVfn+xm5q1N1J3hflfssHnL0y8p10bFykbcY7p1PXkzEmAOwTkdpekEdJgKMoHnp3Tcw5r0prIKJJXr/KkohVuN8VlC7ySYSPe9zA+LkK1WWR64uAMRFV0ENqI11i7cGuZz0lImRR9OAZjpLUySG3iHA9pUjJSw/TY9Nh5WQbyX5zW4CPRORV7MwnAGPMt9MileJJlT1Bv7d6R8y5eKv1RJkw7ph0kd+XdDA02vUUNIa66tLQXtXRO7kFUhyjOHKk1RfqwkndX0X7bD+bKorcIh3/XT1tCpj/jqfkFcVM+5+SQbyyixyiK6x/e/kkVm713tHOwe+PtCiiLYV4RLuegkETs0eFGyc9NlWFYfUDKllz23k9ekaRT2gjP5oCFhLpaDPe46aABUCywewZIlICHGgPLTfG9HwTAqVLuDuzRhNtUfzH4UM7fV50qmyyzeVqPFxPxT6rfbfb1eXwiwsmcuvMZSGLKBsoL/azry2gFkWO4f7vSnXWU3crvdPR+jzbSOqbKyKnADOANVjW1QgRudIY82b6RFOi6Uiw30BpN4qt3IHcIr8kHWyODpoHg4AfFv3sTM8v74WThvXITZQOykv80Jwf6bGFhLshZarrKLqb2OBWMPn6aUp2iXcncKYxZjmAiBwIPAZMTpdgSixeFsWvLj6M4w/o362VTKRF4bPTYTfHv8FmRL8oRWEMQuqC1b2BEy/Jh4K7QiIdLeKdp1SVdu/zWwgfoWSXocWOkgAwxqzA6vek9CLtLoti4U/O5KZzD+aiI4czvG9Ft54XXaF82VEjkrpPRLjtc4eG3FuBHAwKOw0P1aDIPRwXaapjFE6ftJ6QY1+DpElWUcwTkQdF5BT73x+A+YluEJEyEXlfRBaKyBIRucUeHy0ic0RkpYg8bsc+EJFS+/Uq+3x9T95YPtIRNIwZUMlbPziV2opirj55TI/qANz3GgwDa8qS7sR62dEjuXSKpViCJve+IOV28F9dT7lHZWlqK/ydljTR2XzJUghtxpNVFN8AlgDfBr4DLAWu6eSeVmCqMeZw4AjgbBE5FrgduMsYMw7YCVxlX38VsNMYMxa4y75OcdEeCFJbUcyIft2zIKJxu546EgTK4+F8QYwxOfcFceIx6nrKPZyeYamyYp3W99HZfMmSa5/97pCsoigC7jbGfM4Y85/APUBCtW4snPLhYvufAaYCT9rjM4AL7eML7NfY50+TfE0h6CbtgWDM7nI9wV1w15YgUB4P538nYEzOWRRO8F8tityjothyEaXqM+consoUxChy7XuQLMnOOrMAdwSzHKsxYEJExC8iC4AGrPYfnwC7jDEd9iUbACcdZhiwHsA+vxtIvGdmgdEeMDGdW3uC26Jo6+iGorBXUoGgybm0QMeiyLXYigIVtuspVf9zZfaiIVFWYaGTrKIoc1kH2Med+j+MMQFjzBHAcOBo4GCvy+yfXv/vMf4QEZkmIvNEZF5jY2NSwucLHYFgTNO/nuBeTTuBcqfoLpkq7bDrKffSAn2hlMgMC6J0GSdjLVXW4EjblVvRTYuiEEj2L9MsIkcaYz4AEJEpgPfGBx4YY3aJyOvAsVityotsq2E4sMm+bAMwAtggIkVALRDTq8IY8wDwAMCUKVO67ljPYdoDiSugu4q70ntUf+vLctGRw2lu7eALx4zs9H5nsrUsipSJ1Sss32JVrR8+vE+GJVG6SrnjekrR86455QAG1ZRx/qFDunW/25rOte9BsiS7nvou8HcReUtE3gT+Bnwz0Q0iUicifezjcuB0YBkwG7jYvuxK4Fn7+Dn7Nfb510yinhUFSHsgGLO7XE8I2n/eQTWlTD1oEGBlQn3lhNFJ7W3tjlHkmgvnpvMO5rxDhzClvl+mRVG6SKgzQIo+c8V+H58/akRKLJRcc8EmS0KLQkSOAtYbY+aKyEHA14HPAS8Dn3by7CHADLtNuQ94whjzgogsBf4mIrcCHwIP2tc/CPxVRFZhWRKXdfdN5SPH/M+/2NrUykEp3JzGcclOHNq9xsDOV8LkYDD7hLEDOCFOx10lu3Gyk7oTV0sHOfbR7xaduZ5+j2UJABwH/Aj4Fla66wOELYMYjDGLgEke46ux4hXR4y3AJUlJXYBsbbJ2rCtOYZaO09W1uyupkEURNAXxZVGyAyd+tnt/drSbi6yjyE86UxR+Y4wTJ7gUeMAY8xTwlJ3NpPQyqcx6clxP3a0lcMxsq+AuX78iSrYRUhT72jIsiUVEm/E8/Rp05vD224FlgNOA11znNEWgl9jfFggdpzLrybEout8MzXWcp18QJfvINouiEOhssn8MeENEtmFlOb0FICJjseoclDSzdnszn/l/r4deD+sbu91pdzlkqBXvODcV2R4pkUhROqdPRXYpisjvQX5+ExIqCmPML0VkFlZg+hVXFpIPK1ahpJl5a3ZGvI635Wl3GFNXxSf/c263LQr3bbmW9aTkLpNH9cUncNWJYzItClAYi6RO3UfGmPc8xlakRxwlGnfH2F/+50QOS3Hef0+aCkb04S+Eb4uSFfSpKGH1//Zsh8NUoi08lIzjVhSTRvTNoCSxFELXTEXpCvn6LVBFkeW0unLFk93TOhPk60pKUTon/z/8qiiynCZXwC7bFEU6NrpXlFwj4rOfp98DVRRZzoad4ZZa3W2DnC7U9aQokeTr90AVRRazt7WDFxaF97BOZQ1FKnArihS2oFKUnCI/VUMk+vXOYrbvbe3WhkK9ha8A8scVpTM060nJKE37Ozq/KINoZbaiRKWJZ1COdKKKIotpasmOytN4RK6k8vUroihKdkVHlQicjKcHvjSZAwZWZVgaL/J/JaUonVEICyZVFFmM08tm4rBahvZJXY+nVOErAN+sonRGIbQZV9dTFvP3+RuA8EYt2YZ79aS9npRCpRASOVRRZClNLe3MX2s1BKws6Xxb0kxQAHVGitIl8nW9pIoiS/log9XF/dpTDshav6dWZisKEaukfLUuVFFkKXPX7EAEpp2cHa2UvYhUDvn5BVGUziiET74qiizljRWNHD68D30qSjItSlKkcCtvRcld8vR7oIoiS/mkYS+HDa/NtBgJKYSKVEXpjGx1DacSVRRZSGtHgKaWDgZWl2ZalIRoCw9FKYwOBaoospBte9sAGFCV3YpCmwIqitZRKBmicU8rAHVZblFE9rjJ16+IoiiqKLIQR1Fku0Xh00IKRYlyPeXnF0EVRZbQHgjSsKcFgJUNewCoH1CZSZE6x+16ytMviKJ0hkTE6vITVRRZwvSnPuLoX86iPRBk2eY9DOtTTm2Wtu5wKIT2yorSGYXw2VdFkSW8sGgTAG+uaOT5hZs4cFA2douNRNNjFSWSfP0epE1RiMgIEZktIstEZImIfMce7ycir4rISvtnX3tcROQeEVklIotE5Mh0yZaNOK6bq2bMA2BEv4pMipMUPm0KqCjawqOHdADfN8YcDBwLXCciE4DpwCxjzDhglv0a4BxgnP1vGnBfGmXLOqIrm/vmQEW2xrIVJX+Vg5u0KQpjzGZjzAf28R5gGTAMuACYYV82A7jQPr4AeMhYvAf0EZEh6ZIv24jOljDGZEiS5BHVFIoSQb4a1r0SoxCRemASMAcYZIzZDJYyAQbalw0D1rtu22CPFQTuD9iFRwzlyuPrMyZLsuh+FIqSv8rBTdp3uBORKuAp4LvGmKYEecZeJ2KW1SIyDcs1xciRI1MlZsZxT7S/uWxSBiVJnkKoSFWUziiEz35aLQoRKcZSEo8YY562h7c6LiX7Z4M9vgEY4bp9OLAp+pnGmAeMMVOMMVPq6urSJ3wv40y6RTnUhrUQetwoSmcUQvZfOrOeBHgQWGaM+bXr1HPAlfbxlcCzrvEr7OynY4HdjouqEHAsiiJ/7nzS1PWkKJHka2A7na6nE4AvAR+JyAJ77EfAbcATInIVsA64xD73InAusArYB3wljbJlLYNqyjItQtL4CmAlpSidka/KwU3aFIUx5m3iu+9O87jeANelS55sx2/Pug9fdUyGJUke0QRZRVHXk9I7fLqtmcY9rVzzmQNyotDOIaLNeJ5+QRSlMwohqUMVRRZw6h2vAzB+cPa37YhHvq6kFEVRRZFxWjsCoeNxA6szKEnX0R3uFAXcdoS2GVfSwvod+wCYPKovhwytybA0XaMQfLOK0hmF8NlXRZFhPmlsBuAn50/IudWIW1pNj1UUjVEoaWK1rShG12X5JkUeqHJQlMIoPFVFkWFWN+5lQFUpNWXZvUmRF+p6UpTC+OyrosgwG3buZ1T/3EmJdSMazFaUyJ0e81RrqKLIIF/58/u8u3p71m95mgyBYDDTIiiKkiZUUWSQ2csbAagqTXsT37TTHsz+/TMUJR3kqRERgSqKLKAyDxRFR0AtCqUwcfREPisMVRRZgD8P/hc6AmpRKIWJE5fIYz2hiiIbyIfFuLqeFCV/UUWRBQTzYJJV15NS6ORrxhOoosgYizfuDh0HTe4rinZ1PSkFiqMf8ldNqKLICO2BIOf/9u3Q6/GDc6sZoBcdmh6rFCiFUEOkiqKXeX15A+Nuein0elT/Cr56wugMSpQaNJitFDp57HlSRdHbzP64IeL1uIHV+PJg1592jVEoBUrY9ZT73+N4qKLoZYrsXNipBw0EoLm1I5PipAxVFEqhkr/qIYwqil6kcU8r89buZMyASv779AMzLU5K+Me1xwMQyIPMLUXpEXmsMXK/JDiHuO6RD1i4fhcnjh3AIUNr+OapY7n0qBGZFqtHFNsWkmY9KYVKPscmHFRR9CILNuwC4IazxuPzCdefNT7DEvWcsKJQ15NSmBRCZbYqil6gpT3AQTe/DMC0k8dw+Ig+GZYodRT5ra+Hup6UQkV7PSkpoaGpNXQ8om95BiVJPcU+26LQOgpFyVtUUfQCjXvDiuKgITUZlCT1OBZFe4daFEqBoumxSk9ZtrmJy//wXuj1ocNqMyhN6gm5nvKgDYmidAdHQeSz60ljFGlm5qLNtHVYbpkbzhpPWbE/wxKllrqqUq4+aTQXTR6eaVEUJSPks4JwUEWRZtwfoqtPGpM5QdKEiHDTeRMyLYaiZJx81hdpcz2JyJ9EpEFEFrvG+onIqyKy0v7Z1x4XEblHRFaJyCIROTJdcvU2u/a1IwJPfP04SorU06co+UY46yl/VUU6Z66/AGdHjU0HZhljxgGz7NcA5wDj7H/TgPvSKFfa2b2/nfrpM6mfPpO/vreWYX3KOXp0v0yLpShKGshj/RAibYrCGPMmsCNq+AJghn08A7jQNf6QsXgP6CMiQ9IlWzoJBg3n3v1WxNiGnfszJI2iKL1FPuuL3vaFDDLGbAawfw60x4cB613XbbDHco73Vm9n4y5VDIpSKAj5v3NRtjjNvf7EnvmWIjJNROaJyLzGxsY0i9V1npy/IXT8xg2nADCgqiRD0iiKkm4KwfXU21lPW0VkiDFms+1acjZn2AC4u+MNBzZ5PcAY8wDwAMCUKVOyKnn/7ZXbePrDjXxu0jBuOHs8Q2rLee37n6GqTJPLFCVfkaif+UhvWxTPAVfax1cCz7rGr7Czn44FdjsuqlyhIxDk7lkrqKsu5RcXTmRIrdWqY0xdFQOryzIsnaIo6Safs57SttQVkceAU4ABIrIB+ClwG/CEiFwFrAMusS9/ETgXWAXsA76SLrnSxevLG5m7Zie3fPYQKkvVglCUQsFRECaPuxOkbUYzxlwe59RpHtca4Lp0yZJu9rZ2cNvLHwPk/P4SiqJ0Dac+qiOPOyjr0rcHtLQH8PuEi+/7N6sa9jJpZJ+8a9GhKEpiSm1F4bTqyUdUUXSRYNDwl3+vobUjyO0vf8yQ2jI2727h+jMP5LKjR2ZaPEVReplSe3GoFoUS4o2Vjfz8haWh15t3twBwxfH11JQVZ0osRVEyRGkBtOZRRdEFnpq/gVueXxIx9rlJw/jCMSNVSShKgaKKQgGsbIb731jN7XbAusTvo83eI/rXlx6RSdEURckwpUX5H5fMf1XYQ4wx3DNrFbe//DElfh93XXo4935hUqbFUhQlSygtzv9pVC0KD4JBw13/WsHqbc3MXBSu+3vmuhOYMLSGlvZABqVTFCWbUNdTgfLEvPX89rVVEWNHj+7HhKHWftdlxX5uOGs8VVpYpygFTyG4nnSmc/H43HX88KmPACgr9jFhSA1fPXE0Zx0yOKaPy3Wnju19ARVFyTrUoiggOgJBfvpcOKPp+W+eyLhB1RmUSFGUXKBMYxT5z6Nz1vHsgo3M+dTaY+mLx47kFxdMzOsGX4qipA51PRUAP3l2caii8vSDB3HjOQerklAUJWk06ylPWd24l4ffW8fVJ48Ojd192RFccERObqqnKEoGKfVbFsWYAZUZliR9FKSimLWsgT+98yl/eudTAO645HBVEoqidIvaimJuvXAipx08sPOLc5SCVBRfO2k0pcU+5q7ZSWmRj1PG12VaJEVRcpgvHjsq0yKklYJUFCLCFcfVc8Vx9ZkWRVEUJevJ/yiMoiiK0iNUUSiKoigJUUWhKIqiJEQVhaIoipIQVRSKoihKQlRRKIqiKAlRRaEoiqIkRBWFoiiKkhAxxmRahm4jIo3A2m7ePgDYlkJxegOVuXdQmXsHlbl38JJ5lDEm6ZYUOa0oeoKIzDPGTMm0HF1BZe4dVObeQWXuHVIhs7qeFEVRlISoolAURVESUsiK4oFMC9ANVObeQWXuHVTm3qHHMhdsjEJRFEVJjkK2KBRFUZQkKDhFISJni8hyEVklItMzLY8bEfmTiDSIyGLXWD8ReVVEVto/+9rjIiL32O9jkYgcmQF5R4jIbBFZJiJLROQ7OSBzmYi8LyILbZlvscdHi8gcW+bHRaTEHi+1X6+yz9f3tswu2f0i8qGIvJBDMq8RkY9EZIGIzLPHsvbzYcvRR0SeFJGP7c/2cdkss4iMt/++zr8mEfluSmU2xhTMP8APfAKMAUqAhcCETMvlku9k4EhgsWvsV8B0+3g6cLt9fC7wEiDAscCcDMg7BDjSPq4GVgATslxmAars42Jgji3LE+uqznYAAAXMSURBVMBl9vj9wDfs42uB++3jy4DHM/j5+B7wKPCC/ToXZF4DDIgay9rPhy3HDOBr9nEJ0CfbZXbJ7ge2AKNSKXPG3lCG/ojHAf90vb4RuDHTckXJWB+lKJYDQ+zjIcBy+/j3wOVe12VQ9meBM3JFZqAC+AA4BqsgqSj6cwL8EzjOPi6yr5MMyDocmAVMBV6wv+RZLbP9+70URdZ+PoAa4NPov1c2yxwl55nAO6mWudBcT8OA9a7XG+yxbGaQMWYzgP3T2cE9q96L7d6YhLVCz2qZbRfOAqABeBXLytxljOnwkCsks31+N9C/dyUG4DfAD4Cg/bo/2S8zgAFeEZH5IjLNHsvmz8cYoBH4s+3m+6OIVJLdMru5DHjMPk6ZzIWmKMRjLFfTvrLmvYhIFfAU8F1jTFOiSz3Gel1mY0zAGHME1ir9aOBgr8vsnxmXWUTOBxqMMfPdwx6XZo3MLk4wxhwJnANcJyInJ7g2G+QuwnL/3meMmQQ0Y7lt4pENMgNgx6g+C/y9s0s9xhLKXGiKYgMwwvV6OLApQ7Iky1YRGQJg/2ywx7PivYhIMZaSeMQY87Q9nNUyOxhjdgGvY/lp+4hIkYdcIZnt87XAjt6VlBOAz4rIGuBvWO6n35DdMgNgjNlk/2wA/oGlmLP587EB2GCMmWO/fhJLcWSzzA7nAB8YY7bar1Mmc6EpirnAODtbpATLTHsuwzJ1xnPAlfbxlVhxAGf8CjuD4Vhgt2Nm9hYiIsCDwDJjzK9dp7JZ5joR6WMflwOnA8uA2cDFcWR23svFwGvGduz2FsaYG40xw40x9Vif2deMMf9FFssMICKVIlLtHGP5zxeTxZ8PY8wWYL2IjLeHTgOWZrPMLi4n7HaCVMqcqaBLBoM952Jl53wC3JRpeaJkewzYDLRjaf2rsHzLs4CV9s9+9rUC/M5+Hx8BUzIg74lYJusiYIH979wsl/kw4ENb5sXAT+zxMcD7wCos073UHi+zX6+yz4/J8GfkFMJZT1ktsy3fQvvfEuf7ls2fD1uOI4B59mfkGaBvDshcAWwHal1jKZNZK7MVRVGUhBSa60lRFEXpIqooFEVRlISoolAURVESoopCURRFSYgqCkVRFCUhqiiUgkREAlEdNxN2EhaRa0TkihT83jUiMqAb950lIj8Tkb4i8mJP5VCUrlDU+SWKkpfsN1Ybj6QwxtyfTmGS4CSsAruTgXcyLItSYKiiUBQXdpuMx4FT7aEvGGNWicjPgL3GmDtE5NvANUAHsNQYc5mI9AP+hFVktg+YZoxZJCL9sQop67CK38T1u74IfBurlfUc4FpjTCBKnkuxuhyPAS4ABgFNInKMMeaz6fgbKEo06npSCpXyKNfTpa5zTcaYo4F7sXoqRTMdmGSMOQxLYQDcAnxoj/0IeMge/ynwtrEazD0HjAQQkYOBS7Ga5h0BBID/iv5FxpjHCe9RcihWNfkkVRJKb6IWhVKoJHI9Peb6eZfH+UXAIyLyDFaLB7DamVwEYIx5TUT6i0gtlqvoc/b4TBHZaV9/GjAZmGu1zKKccNO2aMZhtVsAqDDG7Eni/SlKylBFoSixmDjHDudhKYDPAjeLyCEkbt3s9QwBZhhjbkwkiFjbhw4AikRkKTDE3kvjW8aYtxK/DUVJDep6UpRYLnX9fNd9QkR8wAhjzGysjYT6AFXAm9iuIxE5BdhmrL053OPnYDWYA6tJ28UiMtA+109ERkULYoyZAszEik/8Cqux3hGqJJTeRC0KpVApt1fmDi8bY5wU2VIRmYO1kLo86j4/8LDtVhLgLmPMLjvY/WcRWYQVzHbaO98CPCYiHwBvAOsAjDFLReTHWLu/+bA6Bl8HrPWQ9UisoPe1wK89zitKWtHusYriws56mmKM2ZZpWRQlW1DXk6IoipIQtSgURVGUhKhFoSiKoiREFYWiKIqSEFUUiqIoSkJUUSiKoigJUUWhKIqiJEQVhaIoipKQ/w9RitpHdaW1PgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1442a6d8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
